Skip to content

CSV import process - improvement

edited January 2013 in Suggestions

Hi there

As a suggestion is it possible to handle imports in 'chunks' of 100-150 users , to get round any potential timeout issues especially on shared hosting , when importing large lists.

I have read some of the other suggestions and I am sorry to say it is not always possible to just increase server timeouts, asides from going out and getting some dedicated tin to do just that.

As an example, PHP Newsletter Mailer is an application that suffered this very problem and got round it by splitting the import process link http://codecanyon.net/item/newsletter-mailer-v13/149365 this is what I am using for another of my larger lists till I am happy with Sendy.

Update: even importing a list of 140 additional users causes a timeout, although when I get back to sendy these 140 are in the NewList I created. The same cant be said when importing 'larger' lists ..it takes a few tries to get the list complete

Thanks

Kb

Comments

  • Hi,

    When importing a CSV file, Sendy checks for duplicate emails, previously bounced emails, invalid emails etc. If you have a large list, it will naturally take longer than if you have a small one.

    Also, the distance between your physical location and your server's location plays a part as well.

    Lastly, Sendy can import CSVs both with or without "quotes".
  • You should import the CSV first and then carry out such data activities - there is no progress bar - you need an AJAX file uploader at min. The CSV is only 1MB but it's taken over a hour so far and it's still not finished.... You should examine things like mail chimp, campaign monitor they don't have such problems... you mention duplicate emails, (there are non in the database already (not the file I'm uploading) there are no previously bounced emails, invalid emails (as it's a fresh install)
  • Are there custom fields in your CSV as well? PM me your CSV and I'll run a test here. Thanks.
  • My suggestion here is to implement something like Delayed job on rails. The user upload the file, that goes into a importing queue. A few seconds later, cron job will take a look in the importing_queue table, and run a script (very long and high processing) that will import that for you.. OR - you can do what I did.. install phpmyadmin, create a table on your database and import all data to it.. then you run SQL to proper format of subscribers list of sendy.
  • There are no custom fields, just email address and name, the file is only around 1MB also... I shoudn't have to install phpmyadmin and create a table and even get to do anything like this. Completely defeats the purpose, clients need to import their own data and would not even know what PHPmyadmin is. Hopefully the makers of sendy can improve import urgently as certainly it's something which is holding back from what on the surface is starting to look very good.
  • Yes, importing CSV seems to be a critical part that requires changes. I have tried reducing my CSV contact list ot 25,000 on each CSV file, however the script does not show any progress at all and eventually receives the the following error:

    Request Timeout
    This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

    If you look at other script like http://codecanyon.net/item/newsletter-mailer-v13/149365 which cost only $14, it handles import of CSV with a progress screen. Screenshot > http://i46.tinypic.com/2a5dvo.jpg

    But the importing speed on Newsletter Mailer v1.3 is really slow, which was why I turned to Sendy hoping to have a faster importing speed. Once the import progress be implemented and with a good import speed, Sendy will be perfect for huge lists users.
  • Hi @KevinB,

    Thanks for your suggestion. I am certainly going to optimize and improve CSV import.

    For now, if your server times out while importing, split your CSV into multiple chunks and import them.

    You mentioned that even importing 140 emails results in a timeout, I think your hosting server is very limiting. If you have a large list and email marketing is important to you or your business, you may want to look at non shared servers. Email marketing applications taxes the server a lot especially if you have a large subscriber base. I'd recommend setting up an Apache/Linux server on Amazon EC2 (just the micro instance will do, with 613MB of memory). I've tested Sendy heavily on it.

    I'm using MediaTemple dedicated virtual server and so far does not have any problems importing or sending.

    That said, I will still improve on the CSV import (improving the speed and working around timeouts).

    Thanks once again.

    Ben

  • @KevinB, put it on AWS on a micro instance and if you have issues you can auto-scale to increase resources to handle it without timing out, then scale down to eliminate the additional expense. I can upload 50,000+ people at one time without issue with AWS (though, I'm not on the free micro instance).

  • I'm jumping in the discussion but the import process it painfully slow, the speed is around 2-3 lines per second. Is this normal?

  • That's been my experience using an AWS micro instance.

    I know sendy does some useful work filtering out invalid emails and checking for duplicates, but it is slow. Alternative, and costlier, solutions are much faster.

  • CSV import can now be imported via a cron job in the just released version 1.1.5. Any timeouts will be handled.

    Note that the larger your CSV, the longer it'll take to import as Sendy not only "imports" your CSV, it "processess" it as well with a number of checks so that invalid emails, duplicates and previously bounced emails won't be imported. But if a timeout occurs because the import takes more than the timeout set by the server/host, Sendy will automatically resume.

    There is also a real time percentage feedback to let you know the progress of the import as well.

This discussion has been closed.