CSV import process - improvement

Comments

• Ben

  December 2012

  Hi,
  
  When importing a CSV file, Sendy checks for duplicate emails, previously bounced emails, invalid emails etc. If you have a large list, it will naturally take longer than if you have a small one.
  
  Also, the distance between your physical location and your server's location plays a part as well.
  
  Lastly, Sendy can import CSVs both with or without "quotes".
• eplatforms

  December 2012

  You should import the CSV first and then carry out such data activities - there is no progress bar - you need an AJAX file uploader at min. The CSV is only 1MB but it's taken over a hour so far and it's still not finished.... You should examine things like mail chimp, campaign monitor they don't have such problems... you mention duplicate emails, (there are non in the database already (not the file I'm uploading) there are no previously bounced emails, invalid emails (as it's a fresh install)
• Ben

  December 2012

  Are there custom fields in your CSV as well? PM me your CSV and I'll run a test here. Thanks.
• thomashs

  December 2012

  My suggestion here is to implement something like Delayed job on rails. The user upload the file, that goes into a importing queue. A few seconds later, cron job will take a look in the importing_queue table, and run a script (very long and high processing) that will import that for you.. OR - you can do what I did.. install phpmyadmin, create a table on your database and import all data to it.. then you run SQL to proper format of subscribers list of sendy.
• eplatforms

  December 2012

  There are no custom fields, just email address and name, the file is only around 1MB also... I shoudn't have to install phpmyadmin and create a table and even get to do anything like this. Completely defeats the purpose, clients need to import their own data and would not even know what PHPmyadmin is. Hopefully the makers of sendy can improve import urgently as certainly it's something which is holding back from what on the surface is starting to look very good.
• djxpire

  December 2012

  Yes, importing CSV seems to be a critical part that requires changes. I have tried reducing my CSV contact list ot 25,000 on each CSV file, however the script does not show any progress at all and eventually receives the the following error:
  
  Request Timeout
  This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.
  
  If you look at other script like http://codecanyon.net/item/newsletter-mailer-v13/149365 which cost only $14, it handles import of CSV with a progress screen. Screenshot > http://i46.tinypic.com/2a5dvo.jpg
  
  But the importing speed on Newsletter Mailer v1.3 is really slow, which was why I turned to Sendy hoping to have a faster importing speed. Once the import progress be implemented and with a good import speed, Sendy will be perfect for huge lists users.
  
• Ben

  January 2013

  Hi @KevinB,

  Thanks for your suggestion. I am certainly going to optimize and improve CSV import.

  For now, if your server times out while importing, split your CSV into multiple chunks and import them.

  You mentioned that even importing 140 emails results in a timeout, I think your hosting server is very limiting. If you have a large list and email marketing is important to you or your business, you may want to look at non shared servers. Email marketing applications taxes the server a lot especially if you have a large subscriber base. I'd recommend setting up an Apache/Linux server on Amazon EC2 (just the micro instance will do, with 613MB of memory). I've tested Sendy heavily on it.

  I'm using MediaTemple dedicated virtual server and so far does not have any problems importing or sending.

  That said, I will still improve on the CSV import (improving the speed and working around timeouts).

  Thanks once again.

  Ben
• vincentpolisi

  January 2013

  @KevinB, put it on AWS on a micro instance and if you have issues you can auto-scale to increase resources to handle it without timing out, then scale down to eliminate the additional expense. I can upload 50,000+ people at one time without issue with AWS (though, I'm not on the free micro instance).
• ozgurakman

  February 2013

  I'm jumping in the discussion but the import process it painfully slow, the speed is around 2-3 lines per second. Is this normal?
• Steve

  February 2013

  That's been my experience using an AWS micro instance.

  I know sendy does some useful work filtering out invalid emails and checking for duplicates, but it is slow. Alternative, and costlier, solutions are much faster.
• Ben

  February 2013

  CSV import can now be imported via a cron job in the just released version 1.1.5. Any timeouts will be handled.

  Note that the larger your CSV, the longer it'll take to import as Sendy not only "imports" your CSV, it "processess" it as well with a number of checks so that invalid emails, duplicates and previously bounced emails won't be imported. But if a timeout occurs because the import takes more than the timeout set by the server/host, Sendy will automatically resume.

  There is also a real time percentage feedback to let you know the progress of the import as well.