Strange behavior when importing lists
First off I love Sendy, Ben you've done a fantastic job in putting this together and I can't wait to see what you have in store for the future.
With that said, I'm seeing a bit of strange behavior when importing lists. This list in particular is rather large, so I've had to split the import into 3 separate CSV files. This has worked fine for the most part, but last night when importing the 3rd file, the server load spiked to 20 and froze up everything. Looking at the running MySQL processes I saw there were ~30 different import processes going on at once (normally I'd only see a single import process at a time). I ended up restarting MySQL to get things back to normal, but now I'm not able import anything to this list as I just see an odd completion percentage when attempting to import additional files.
http://cl.ly/image/0k2a3w0X0o29
And this is what I see when viewing the MySQL processes.
http://cl.ly/image/1t310i0j2133
I sent out a test mailer to this list and everything worked fine.
I'm not really concerned with the ability to add addresses to this list in particular (as this was done as a test list), but it would be great to know why the load suddenly spiked and locked everything up so I can avoid it in the future.
Also, is it expected behavior for it to take longer to update/add addresses to larger lists? I.E. Importing the first 10,000 addresses takes just a few seconds, but adding those same addresses to a list that already contains say 100,000 users will take 10-20 minutes. This is even true when importing a list for the firsts time (the first ~30,000 names get added quickly, but then after that process slows down quite a bit).
Sendy is the only thing running on this server, and has zero outside traffic at this point. Server load normally stays in the 1.0 - 1.5 range importing.
Server info
- LiquidWeb 12GB SSD Cloud server
- 12 GB RAM
- 4 x Xeon(R) CPU E3-1220 V2 @ 3.10GHz
- PHP 5.4.14
- MySQL 5.5.30-cll
- Apache 2.2.24
Lastly, if you (or anyone else) would like a large test list (can be used for import or send rate testing) here is one I've put together that uses the Amazon SES Mailbox Simulator. Sending to this list is 100% sandboxed, and it won't count against your sending limits, nor will you be charged for any of the test emails.
http://cl.ly/2E0x272c0K1B
Thanks!
Comments
Hi James,
I have never seen the load spike before so I don't have an explanation for why this is happening.
In future, you could just upload the entire CSV without splitting it into 3s if you are using a cron job to import CSVs. Any timeouts will be handled as the cron job will continuously check if importing needs resuming.
There are a couple of indexes in the subscribers table so adding another tens of thousands of records into a database filled with hundreds of thousands of records will be slower.
Indexes were added in later versions of Sendy as a lot of users wanted indexes to be added to improve sending speed.
Thanks Ben. When I tried doing the CSV in one fell swoop Sendy gave me a message in a red box that said something 'Please upload a valid CSV'. Splitting up the same file worked just fine the 2nd time around.
Do you think it's possible the MySQL got stuck, then a new process started over and over again the next time the cron ran?
Here's a screenshot that shows a huge spike in MySQL processes around the time the server load spiked.
http://cl.ly/Oetf
Cheers!
Hi James,
To be honest, I'm not too sure, I didn't experience this when I tested. Can you try again and let me know if this happens again? It may be an one off incident.
Thanks.
P.S. I received your newsletter (sent via Sendy) for your new product, Infuse. Congrats on your launch!
Ben
Thanks Ben. Everything did work the 2nd time around, so it may just have been a one-time fluke.
Sendy worked flawlessly for the Infuse launch, so thanks again for such a great product.
I'm really glad to hear. Congrats once again for your launch.
Ben