One of the major pains of shared hosting migration is user email on the system. This process enables a fairly easy method of migration using an application on the command line of a CentOS server (or any other linux host) as an intermediary.
[hr top=”0″ bottom=”18″ /]
Log into the server you wish to be the migrator:
ssh user@YOURLINUXBOX.FQDN.COM
Next we must install imapsync, the next two commands are dependent on your architecture, choose one:
RHEL based distro:
sudo yum -y install imapsync
Debian based distro:
sudo apt-get -y install imapsync
It will install the perl dependencies and the imapsync CLI application.
[hr top=”0″ bottom=”18″ /]
Next we’ll issue the sync commands.
Here are some of the options you can use (from the manpage):
[hr top=”0″ bottom=”18″ /]
usage: /usr/bin/imapsync [options]
Several options are mandatory.
--dry : Makes imapsync doing nothing, just print what would
be done without --dry.
--host1 : Source or "from" imap server. Mandatory.
--port1 : Port to connect on host1. Default is 143, 993 if --ssl1
--user1 : User to login on host1. Mandatory.
--showpasswords : Shows passwords on output instead of "MASKED".
Useful to restart a complete run by just reading the log.
--password1 : Password for the user1.
--host2 : "destination" imap server. Mandatory.
--port2 : Port to connect on host2. Default is 143, 993 if --ssl2
--user2 : User to login on host2. Mandatory.
--password2 : Password for the user2.
--passfile1 : Password file for the user1. It must contain the
password on the first line. This option avoids to show
the password on the command line like --password1 does.
--passfile2 : Password file for the user2. Contains the password.
--ssl1 : Use a SSL connection on host1.
--ssl2 : Use a SSL connection on host2.
--tls1 : Use a TLS connection on host1.
--tls2 : Use a TLS connection on host2.
--timeout : Connections timeout in seconds. Default is 120.
0 means no timeout.
--authmech1 : Auth mechanism to use with host1:
PLAIN, LOGIN, CRAM-MD5 etc. Use UPPERCASE.
--authmech2 : Auth mechanism to use with host2. See --authmech1
--authuser1 : User to auth with on host1 (admin user).
Avoid using --authmech1 SOMETHING with --authuser1.
--authuser2 : User to auth with on host2 (admin user).
--proxyauth1 : Use proxyauth on host1. Requires --authuser1.
Required by Sun/iPlanet/Netscape IMAP servers to
be able to use an administrative user.
--proxyauth2 : Use proxyauth on host2. Requires --authuser2.
--authmd51 : Use MD5 authentification for host1.
--authmd52 : Use MD5 authentification for host2.
--domain1 : Domain on host1 (NTLM authentication).
--domain2 : Domain on host2 (NTLM authentication).
--folder : Sync this folder.
--folder : and this one, etc.
--folderrec : Sync this folder recursively.
--folderrec : and this one, etc.
--folderfirst : Sync this folder first. --folderfirst "Work"
--folderfirst : then this one, etc.
--folderlast : Sync this folder last. --folderlast "[Gmail]/All Mail"
--folderlast : then this one, etc.
--nomixfolders : Do not merge folders when host1 is case sensitive
while host2 is not (like Exchange). Only the first
similar folder is synced (ex: Sent SENT sent -> Sent).
--skipemptyfolders : Empty host1 folders are not created on host2.
--include : Sync folders matching this regular expression
--include : or this one, etc.
in case both --include --exclude options are
use, include is done before.
--exclude : Skips folders matching this regular expression
Several folders to avoid:
--exclude 'fold1|fold2|f3' skips fold1, fold2 and f3.
--exclude : or this one, etc.
--regextrans2 : Apply the whole regex to each destination folders.
--regextrans2 : and this one. etc.
When you play with the --regextrans2 option, first
add also the safe options --dry --justfolders
Then, when happy, remove --dry, remove --justfolders.
Have in mind that --regextrans2 is applied after prefix
and separator inversion.
--tmpdir : Where to store temporary files and subdirectories.
Will be created if it doesn't exist.
Default is system specific, Unix is /tmp but
it's often small and deleted at reboot.
--tmpdir /var/tmp should be better.
--pidfile : The file where imapsync pid is written.
--pidfilelocking : Abort if pidfile already exists. Usefull to avoid
concurrent transfers on the same mailbox.
--nolog : Turn off logging on file
--logfile : Change the default logfile pathname and filename.
--prefix1 : Remove prefix to all destination folders
(usually INBOX. or INBOX/ or an empty string "")
you have to use --prefix1 if host1 imap server
does not have NAMESPACE capability, all other
cases are bad.
--prefix2 : Add prefix to all host2 folders. See --prefix1
--sep1 : Host1 separator in case NAMESPACE is not supported.
--sep2 : Host2 separator in case NAMESPACE is not supported.
--skipmess : Skips messages maching the regex.
Example: 'm/[\x80-ff]/' # to avoid 8bits messages.
--skipmess is applied before --regexmess
--skipmess : or this one, etc.
--disarmreadreceipts : Disarms read receipts (host2 Exchange issue)
--regexmess : Apply the whole regex to each message before transfer.
Example: 's/\000/ /g' # to replace null by space.
--regexmess : and this one, etc.
--regexflag : Apply the whole regex to each flags list.
Example: 's/"Junk"//g' # to remove "Junk" flag.
--regexflag : and this one, etc.
--delete : Deletes messages on host1 server after a successful
transfer. Option --delete has the following behavior:
it marks messages as deleted with the IMAP flag
\Deleted, then messages are really deleted with an
EXPUNGE IMAP command.
--delete2 : Delete messages in host2 that are not in
host1 server. Useful for backup or pre-sync.
--delete2duplicates : Delete messages in host2 that are duplicates.
Works only without --useuid since duplicates are
detected with an header part of each message.
--delete2folders : Delete folders in host2 that are not in host1 server.
For safety, first try it like this (it is safe):
--delete2folders --dry --justfolders --nofoldersizes
--delete2foldersonly : Deleted only folders matching regex.
Example: --delete2foldersonly "/^Junk$|^INBOX.Junk$/"
--delete2foldersbutnot : Do not delete folders matching regex.
Example: --delete2foldersbutnot "/Tasks$|Contacts$|Foo$/"
--noexpunge : Do not expunge messages on host1.
Expunge really deletes messages marked deleted.
Expunge is made at the beginning, on host1 only.
Newly transferred messages are also expunged if
option --delete is given.
No expunge is done on host2 account (unless --expunge2)
--expunge1 : Expunge messages on host1 after messages transfer.
--expunge2 : Expunge messages on host2 after messages transfer.
--uidexpunge2 : uidexpunge messages on the host2 account
that are not on the host1 account, requires --delete2
--nomixfolders : Avoid merging folders that are considered different on
host1 but the same on destination host2 because of
case sensitivities and insensitivities.
--syncinternaldates : Sets the internal dates on host2 same as host1.
Turned on by default. Internal date is the date
a message arrived on a host (mtime).
--idatefromheader : Sets the internal dates on host2 same as the
"Date:" headers.
--maxsize : Skip messages larger (or equal) than bytes
--minsize : Skip messages smaller (or equal) than bytes
--maxage : Skip messages older than days.
final stats (skipped) don't count older messages
see also --minage
--minage : Skip messages newer than days.
final stats (skipped) don't count newer messages
You can do (+ are the messages selected):
past|----maxage+++++++++++++++>now
past|+++++++++++++++minage---->now
past|----maxage+++++minage---->now (intersection)
past|++++minage-----maxage++++>now (union)
--search : Selects only messages returned by this IMAP SEARCH
command. Applied on both sides.
--search1 : Same as --search for selecting host1 messages only.
--search2 : Same as --search for selecting host2 messages only.
--search CRIT equals --search1 CRIT --search2 CRIT
--exitwhenover : Stop syncing when total bytes transferred reached.
Gmail per day allows 2500000000 down 500000000 upload.
--maxlinelength : skip messages with a line length longer than bytes.
RFC 2822 says it must be no more than 1000 bytes.
--useheader : Use this header to compare messages on both sides.
Ex: Message-ID or Subject or Date.
--useheader and this one, etc.
--subscribed : Transfers subscribed folders.
--subscribe : Subscribe to the folders transferred on the
host2 that are subscribed on host1. On by default.
--subscribeall : Subscribe to the folders transferred on the
host2 even if they are not subscribed on host1.
--nofoldersizes : Do not calculate the size of each folder in bytes
and message counts. Default is to calculate them.
--nofoldersizesatend : Do not calculate the size of each folder in bytes
and message counts at the end. Default is on.
--justfoldersizes : Exit after having printed the folder sizes.
--syncacls : Synchronises acls (Access Control Lists).
--nosyncacls : Does not synchronize acls. This is the default.
Acls in IMAP are not standardized, be careful.
--usecache : Use cache to speedup.
--nousecache : Do not use cache. Caveat: --useuid --nousecache creates
duplicates on multiple runs.
--useuid : Use uid instead of header as a criterium to recognize
messages. Option --usecache is then implied unless
--nousecache is used.
--debug : Debug mode.
--debugcontent : Debug content of the messages transfered.
--debugflags : Debug flags.
--debugimap1 : IMAP debug mode for host1. imap debug is very verbose.
--debugimap2 : IMAP debug mode for host2.
--debugimap : IMAP debug mode for host1 and host2.
--tests : Run non-regression tests.
--testslive : Run a live test with test1.lamiral.info imap server.
Useful to check the basics. Needs internet connexion.
--version : Print software version.
--noreleasecheck : Do not check for new imapsync release (a http request).
--releasecheck : Check for new imapsync release (a http request).
--justconnect : Just connect to both servers and print useful
information. Need only --host1 and --host2 options.
--justlogin : Just login to both host1 and host2 with users
credentials, then exit.
--justfolders : Do only things about folders (ignore messages).
--help : print this help.
Example: to synchronize imap account "test1" on "test1.lamiral.info"
to imap account "test2" on "test2.lamiral.info"
with test1 password "secret1"
and test2 password "secret2"
/usr/bin/imapsync \
--host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
[hr top=”0″ bottom=”18″ /]
OK, thats a lot of options. For this situation let’s assume we’re migrating from one shared hosting to another (e.g. bluehost->inmotionhosting)
You should create the account you wish to sync on the target host. Use your cpanel to create the same user and password at the target.
In this example that’s USERNAME@DOMAIN.COM with the password ‘PASSWORD’.
Find out what your shared host uses for IMAP connections. Also make note whether they offer Secure IMAP (over SSL or TLS) you’ll need that info for the next command.
Once you’ve created the target account and imapsync is installed on the intermediary server you can dry-run sync the imap trees with one command:
imapsync --dry --ssl2 --host1 YOURIMAP.bluehost.com --user1 USERNAME@DOMAIN.com --password1 'PASSWORD' --host2 YOURSECUREIMAP.inmotionhosting.com --user2 USERNAME@DOMAIN.com --password2 'PASSWORD'
In order, here’s the breakdown of the command:
[hr top=”0″ bottom=”18″ /]
–dry creates a dry run, this performs a non-destructive test, if you see the output you’re looking for you can continue. Otherwise it protects you from a bad sync.
–ssl2 says that the new host (host2) is using ssl, you can specify the same flag for host1 by adding –ssl1
–host1 YOURIMAP.bluehost.com specifies the name of the host that holds the mailbox to be migrated. Verify whether or not you need ssl/tls for this connection.
–user1 USERNAME@DOMAIN.com this will be the username to log in to host1’s email box
–password1 ‘PASSWORD’ you should add the single quotes especially if there are special characters in the password.
–host2 YOURSECUREIMAP.inmotionhosting.com sets the second host, or the target of the migration. also, verify ssl/tls and specify in the initial flags if needed.
–user2 USERNAME@DOMAIN.com this in most cases will be the exact same username and password, but created on the new host (host2)
–password2 ‘PASSWORD’ the password for the target account on host2
Update that command string with the required flags and info, and run it. You should see output like this:
[hr top=”0″ bottom=”18″ /]
imapsync --dry --ssl2 --host1 YOURIMAP.bluehost.com --user1 USER@DOMAIN.com --password1 'PASSWORD' --host2 YOURSECUREIMAP.inmotionhosting.com --user2 USER@DOMAIN.com --password2 'PASSWORD' Transfer started at Wed Nov 4 08:27:10 2015 PID is 24597 Log file is LOG_imapsync/2015_11_04_08_27_10_USER@DOMAIN.com.txt ( to change it, use --logfile filepath ; or use --nolog to turn off logging ) $RCSfile: imapsync,v $ $Revision: 1.637 $ $Date: 2015/04/01 01:36:37 $ Here is a [linux] system (Linux YOURLINUXBOX.FQDN.com 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 22:00:00 UTC 2015 x86_64) With perl 5.10.1 Mail::IMAPClient 3.34 Command line used: /usr/bin/imapsync --dry --ssl2 --host1 YOURIMAP.bluehost.com --user1 USER@DOMAIN.com --password1 MASKED --host2 YOURSECUREIMAP.inmotionhosting.com --user2 USER@DOMAIN.com --password2 MASKED Temp directory is /tmp ( to change it use --tmpdir dirpath ) PID file is /tmp/imapsync.pid ( to change it use --pidfile filepath ; to avoid it use --pidfile "" ) Modules version list: Mail::IMAPClient 3.34 IO::Socket 1.31 IO::Socket::IP ? IO::Socket::INET 1.31 IO::Socket::SSL 1.31 Net::SSLeay 1.35 Compress::Zlib 2.021 Digest::MD5 2.39 Digest::HMAC_MD5 1.01 Digest::HMAC_SHA1 1.01 Term::ReadKey 2.30 File::Spec 3.3 Time::HiRes 1.9721 Unicode::String 2.09 IO::Tee 0.64 File::Copy::Recursive 0.38 Authen::NTLM 1.09 URI::Escape 3.29 Data::Uniqid 0.12 ( use --no-modules_version to turn off printing this Perl modules list ) Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1. Info: will try to use LOGIN authentication on host1 Info: will try to use LOGIN authentication on host2 Info: imap connexions timeout is 120 seconds Host1: IMAP server [YOURIMAP.bluehost.com] port [143] user [USER@DOMAIN.com] Host2: IMAP server [YOURSECUREIMAP.inmotionhosting.com] port [993] user [USER@DOMAIN.com] Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. Host1: YOURIMAP.bluehost.com says it has CAPABILITY for AUTHENTICATE LOGIN Host1: success login on [YOURIMAP.bluehost.com] with user [USER@DOMAIN.com] auth [LOGIN] Host2: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. Host2: YOURSECUREIMAP.inmotionhosting.com says it has NO CAPABILITY for AUTHENTICATE LOGIN Host2: success login on [YOURSECUREIMAP.inmotionhosting.com] with user [USER@DOMAIN.com] auth [LOGIN] Host1: state Authenticated Host2: state Authenticated Host1 capability: IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA Host2 capability: IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION Host1: separator given by NAMESPACE: [.] Host2: separator given by NAMESPACE: [.] Host1: prefix given by NAMESPACE: [INBOX.] Host2: prefix given by NAMESPACE: [INBOX.] Host1 separator and prefix: [.][INBOX.] Host2 separator and prefix: [.][INBOX.] ++++ Listing folders All foldernames are presented between brackets like [X] where X is the foldername. When a foldername contains non-ASCII characters it is presented in the form [X] = [Y] where X is the imap foldername you have to use in command line options and Y is the uft8 output just printed for convenience, to recognize it. Host1 folders list: [INBOX] [INBOX.Drafts] [INBOX.Junk] [INBOX.Sent] [INBOX.Trash] Host2 folders list: [INBOX] [INBOX.Drafts] [INBOX.Junk] [INBOX.Sent] [INBOX.Trash] Folders sizes before the synchronization. You can remove foldersizes listings by using "--nofoldersizes" and "--nofoldersizesatend" but then you will also loose the ETA (Estimation Time of Arrival) given after each message copy. ++++ Calculating sizes on Host1 Host1 folder [INBOX] Size: 538025884 Messages: 8023 Biggest: 30508221 Host1 folder [INBOX.Drafts] Size: 0 Messages: 0 Biggest: 0 Host1 folder [INBOX.Junk] Size: 3748 Messages: 1 Biggest: 3748 Host1 folder [INBOX.Sent] Size: 108469842 Messages: 170 Biggest: 30507437 Host1 folder [INBOX.Trash] Size: 87568 Messages: 8 Biggest: 30073 Host1 Nb messages: 8202 messages Host1 Total size: 646587042 bytes (616.633 MiB) Host1 Biggest message: 30508221 bytes (29.095 MiB) Host1 Time spent: 4.3 seconds ++++ Calculating sizes on Host2 Host2 folder [INBOX] Size: 0 Messages: 0 Biggest: 0 Host2 folder [INBOX.Drafts] Size: 0 Messages: 0 Biggest: 0 Host2 folder [INBOX.Junk] Size: 0 Messages: 0 Biggest: 0 Host2 folder [INBOX.Sent] Size: 0 Messages: 0 Biggest: 0 Host2 folder [INBOX.Trash] Size: 0 Messages: 0 Biggest: 0 Host2 Nb messages: 0 messages Host2 Total size: 0 bytes (0.000 KiB) Host2 Biggest message: 0 bytes (0.000 KiB) Host2 Time spent: 0.1 seconds ++++ Looping on each folder [INBOX] -> [INBOX] Subscribing to folder INBOX on destination server msg INBOX/1 copying to INBOX (not really since --dry mode) msg INBOX/2 copying to INBOX (not really since --dry mode) msg INBOX/3 copying to INBOX (not really since --dry mode) msg INBOX/4 copying to INBOX (not really since --dry mode) ### TRUNCATED HERE ###
Towards the end there, you see it starts copying messages, but it’s simulated because of dry run.
If you feel the output matches your desired outcome, you can stop the process with CTRL+C and remove –dry from the command to begin sync.
Once the sync has completed, verify the new account by logging into webmail. If everything looks good, you can change your DNS settings to point to the new server.
Because of propagation, be sure to monitor the old account for a few days before deleting it, some mail services may cache the DNS lookup and deliver to the old box. This is unfortunately unavoidable.