Sendy still sending more than once to same subscriber

thomashs

Ben. Looks like sendy still doesn't work properly. Today it sent more than one time to lots of subscribers again! I checked the recipients count and stil over the list count. My guess is that somehow, your script of filling the queue table before the campaign is considered good to go is not finishing before another cron call is made. So it fills the queue forever... As I said before, my queue table was going over millions of records with a sigle campaign with 15.000 subscribers. That's 2 problems: a timeout issue, that application logic should handle.. and a logic issue, to avoid sending to same contact twice (or more!!). My suggestion is: write a PYTHON or RUBY daemon that handles the sending queue, either to insert the email list into the queue and actually send them..

I'm running sendy on a Amazon EC2 micro instance. Disk space is ok.
Which variables, like timeout setting or other, do I have to check?

derivativemedia

FWIW, I am having the same issue and I rolled back to sendy 1.1.1.3 as the last working release where this isn't an issue -- and dropped the queue table before rolling back.

Ben

@thomashs

First of all, I want to say, thanks for replying and helping others out. I saw your replies on other threads.

Right after reading this thread, I sent a test email to 10,000 subscribers. I am using EC2's micro instance (same as you), on Apache / Linux. If you're wondering how I did a test send to so many emails, I set one of my email address as 'catch all', then created a list with emails like, test1@email.com, test2@email.com and so on until test10000@email.com.

Sendy sent successfully to 10,000 emails:


Amazon SES metrics says 10,000 emails were sent, no more and no less:


In 1.1.3.1, I've already made doubly sure no emails are added to the queue, twice.

1. The queue table is checked for existing email before adding an email to the queue
2. When an email is sent, it's marked as 'sent' as a second safety precautionary check so that this status will be checked against if the script happens to want to send the same email again for whatever reason it may be

As you can see I was able to send 10,000 emails using the same setup as you (EC2 micro instance) and using the latest Sendy version 1.1.3.1, there should be no issue with Sendy.

What I can suggest to you now is this:

1. Make sure you don't have any currently sending campaigns
2. Delete everything in your 'queue' table by uploading this script to the root of your installation and run it
3. Re-upload all files from version 1.1.3.1 to your installation (excluding /includes/config.php and /uploads/ folder)
4. Make sure you're not using any subversion software to upgrade Sendy as this proves to be problematic from experiences of some users
5. Make sure you're not using any caching software on your server as well
6. Also, make sure you only have one cron job executing the scheduled.php script and the timing is "every 5 mins" like so → */5 * * * * php /path/to/scheduled.php > /dev/null 2>&1

For points 4, 5 and 6, I just want to make sure.

Which variables, like timeout setting or other, do I have to check?

Sendy automatically takes care of timeouts by automatically resuming sending if your server times out the sending script. This was highly requested by many previously and was introduced in version 1.1.1.

Ben

@derivativemedia

I deleted your previous thread and bring it here so I can answer you in one post.

Since having that queue issue, I rolled back to 1.1.1.3 which is stable and working normally for me. I saw you rolled out 1.1.3.1 last night. If I download that install, are ALL files in that release updated so that I can go from 1.1.1.3 to 1.1.3.1 without upgrading sequentially with the other several release in between?

Yes, you don't have to update sequentially. Just download 1.1.3.1. Then upload and replace everything on your server (except /includes/config.php and /uploads/ folder).

FWIW, I am having the same issue and I rolled back to sendy 1.1.1.3 as the last working release where this isn't an issue -- and dropped the queue table before rolling back.

Your issue as I previously understood was that your database's size increased by several MBs when using 1.1.3. This is resolved in 1.1.3.1 where the query_str that holds the encrypted email message is emptied each time an email is sent. This reduces your database size dramatically and increases sending speed at the time of sending.

And, yes the queue table will be filled up with rows during sending. This is normal. They will not be deleted until all emails are sent unlike previous versions where each row is deleted after each email is sent. This is a precautionary feature (in response to @thomashs) so that the script can check the sent column to see if an email has already been sent.

In 1.1.3, the query_str column that holds the encrypted email message wasn't emptied, hence your database size grew by a lot.

1.1.3.1 empties the query_str column for every email marked as 'sent'. So the database size will not grow dramatically and as a result increases sending speed at the time of sending.

Ben

And I've verified the test send of 10,000 emails was sent to exactly 10,000 emails:

thomashs

Did you try to send to 15.000, 20.000 ?

Ergun

I have sent today to a list of 56.667 subscribers. Sendy shows it sent to 56.989 subscribers. I'm using 1.1.3.1. Running on aws m1.small instance.

Ben

@thomashs

I just sent a test email to 26,376 emails.

Here's what Amazon SES metrics displays:



Note that I've sent 10,000 emails previously for the previous test. If you look at the graph, it sent around 26000++ emails starting from 10,000 onwards. The exact quota number is not updated yet and emails are still trickling in from Amazon.

Ben

@Ergun Does your installation show the campaign as "Sent" and the figure is "56989"? If so, emails have been sent to "56989". Was any new subscribers added to any of your lists when the campaign was sending?

Ergun

yes and no. Yes to campaign is finished sending, no new subscribers, ses shows 56988 sent. There must be some duplicates sent??

Ben

Did you send any emails prior to this campaign?

Ben

Was your SES quota zero before you sent this campaign?

Ergun

@Ben I sent yesterday. I waited today until my quota was down to zero again.

jpansel

I'm starting to worry... Is it difficult to roll back to v.1.1.1.3 ? Is it only a question of replacing the files, or it there something to do on the database as well ?

Ben

@thomashs @Ergun @jpansel I've sent 2 test emails, one to 10,000 and another to 26,376 and there were no duplicates. And I've already implemented lots of checks like checking if emails exists between lists, checking if email exists in queue before adding and checking if the email in the queue was marked as sent. There is no way duplicates will be sent. I've done many tests before this and there were no duplicates sent as well. There were absolutely no duplicates sent on all my tests.

Ben

My second test sent an email to 26,376 emails.

I've received every single email in my 'catch-all' gmail account. I waited a few hours to make sure no more emails are coming in. Here's the result:





Amazon SES console also shows the correct number:

Before 2nd test was sent:



After 2nd test was sent:



The difference between the 2 figures is 26,379. The 2nd test was to 26,376 emails. So between 26,379 and 26,376 there are 3 extra emails. 1 email was for the 'sending complete email notification' and 2 emails were for 2 autoresponder emails sent to 2 users who signed up for our newsletter on our website. The final figure works out to be exactly what was sent by Sendy for the 2nd test.

Sending is as accurate as it can get. It sends nothing more and nothing less as far as my test goes.

thomashs

Did you try to force timeouts / interruptions?

Ben

Hi @thomashs. I have previously tested auto resume before releasing it. For the sake of this discussion, I did another test send to 15,000. After a certain number (around 9000+), I stopped my Amazon instance to simulate a timeout.

Then started the instance again, started Apache and mySQL. Auto resume kicks in and eventually finished sending.

I did the test yesterday and I have received all 15,000 emails, not any more or less.

Here's a screenshot of all 15,000 emails received:

jpansel

Yesterday I sent a campaign to 17 000 email addresses. The host is really cheap, very low time out, and the mysql database went offline a few times. I guess Sendy was doing too many queries for this cheap hosting.

But I never had to click resend, and at the end everything went well, without any duplicates :-)

Now I've got to find a new host...

derivativemedia

@jpansel -- I'd highly recommend an EC2 instance t1.micro or m1.small