How to Set Up Email Server on a Dedicated Hosting Environment

To set up an email server on a dedicated server, assign a proper hostname (mail.example.com), install and secure Postfix (SMTP) and Dovecot (IMAP/POP3) with TLS, open mail ports, configure DNS (MX, SPF, DKIM, DMARC) and rDNS, enable spam filtering, create mailboxes, and verify deliverability with major providers.

Setting up an email server on a dedicated hosting environment gives you full control, better privacy, and brand trust—but it also demands careful configuration for security and deliverability. In this guide, you’ll learn how to set up email server on a dedicated server using Postfix and Dovecot, configure DNS correctly, and pass modern anti-spam checks.

What You Need Before You Begin

Server, OS, and Hostname (FQDN)

Use a clean dedicated server (Ubuntu 22.04/24.04 LTS or RHEL/Rocky 9 recommended) with at least 2 vCPU, 4 GB RAM, and SSD. Set a permanent hostname to a fully qualified domain name, typically mail.example.com, and point an A/AAAA record to your server’s IP(s).

Clean IP and Reverse DNS

Email deliverability hinges on IP reputation. Start with a clean IP (not on blacklists), then set a PTR (reverse DNS) to match your mail hostname (e.g., 203.0.113.10 → mail.example.com). Many providers require a request to support to set rDNS; YouStable support can set PTR on your dedicated IPs on request.

Open Ports and Provider Policies

Ensure your provider allows outbound SMTP (port 25). Open these ports in your firewall: 25 (SMTP), 465 (SMTPS), 587 (Submission), 110/995 (POP3/POP3S), 143/993 (IMAP/IMAPS), and 4190 (Sieve, optional). Some clouds block 25 by default—request an unblock or use a smart host relay if needed.

Two Paths: Control Panel or Manual Stack

If You Prefer a Control Panel

cPanel, Plesk, and DirectAdmin automate most email tasks (mailboxes, SPF/DKIM, webmail). They’re ideal if you want speed over manual control. If you choose a panel, much of the SMTP/IMAP configuration is handled for you. You’ll still need correct DNS, rDNS, and security policies.

If You Want Full Control (Our Stack)

We’ll use Postfix (MTA) for sending/receiving and Dovecot for IMAP/POP3. For spam filtering and authentication, we’ll cover Rspamd or SpamAssassin with OpenDKIM/OpenDMARC. This approach is flexible and production-grade when configured properly.

Step-by-Step: Install and Configure Your Email Server

1) Update, Set Timezone, and Configure Firewall

Accurate time and security are prerequisites. Replace example.com with your domain as needed.

# Ubuntu
sudo apt update && sudo apt -y upgrade
sudo timedatectl set-timezone UTC

# RHEL/Rocky
sudo dnf -y update
sudo timedatectl set-timezone UTC

# UFW firewall (Ubuntu)
sudo ufw allow 22/tcp
sudo ufw allow 25,465,587/tcp
sudo ufw allow 110,995,143,993/tcp
sudo ufw allow 4190/tcp
sudo ufw enable

2) Set Hostname and DNS Records

Set your mail hostname (must match rDNS):

sudo hostnamectl set-hostname mail.example.com

In your DNS zone, add:

• A record: mail.example.com → 203.0.113.10
• MX record: example.com → mail.example.com (priority 10)
• TXT SPF: “v=spf1 a mx ip4:203.0.113.10 ~all”

We’ll add DKIM and DMARC after configuring signing.

3) Install Postfix and Dovecot

# Ubuntu
sudo apt -y install postfix dovecot-imapd dovecot-pop3d

# RHEL/Rocky
sudo dnf -y install postfix dovecot

When prompted, choose “Internet Site” and set mail.example.com as system mail name.

4) Obtain TLS Certificates (Let’s Encrypt)

Encrypt SMTP, IMAP, and POP3. Ensure mail.example.com resolves to your server first.

# Ubuntu (snap)
sudo snap install --classic certbot
sudo certbot certonly --standalone -d mail.example.com --agree-tos -m admin@example.com --non-interactive

# Cert paths (commonly)
# /etc/letsencrypt/live/mail.example.com/fullchain.pem
# /etc/letsencrypt/live/mail.example.com/privkey.pem

5) Postfix: Core Configuration

Edit /etc/postfix/main.cf and set minimum secure options. Replace example values.

myhostname = mail.example.com
myorigin = /etc/mailname
mydestination = $myhostname, localhost
inet_interfaces = all
inet_protocols = ipv4
mynetworks = 127.0.0.0/8
message_size_limit = 30720000

# TLS
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtp_tls_security_level = may
smtpd_tls_security_level = may
smtpd_tls_loglevel = 1

# Authentication (via Dovecot)
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_recipient_restrictions = permit_sasl_authenticated, reject_unauth_destination

# Submission tuning
submission_recipient_restrictions = permit_sasl_authenticated, reject
smtpd_relay_restrictions = permit_sasl_authenticated, reject_unauth_destination

Enable Submission (587) and SMTPS (465) in /etc/postfix/master.cf:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes

Reload Postfix:

sudo systemctl enable --now postfix
sudo systemctl reload postfix

6) Dovecot: IMAP/POP3 and Auth

Configure SSL and authentication in Dovecot. Edit /etc/dovecot/dovecot.conf or conf.d files:

protocols = imap pop3 lmtp
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

auth_mechanisms = plain login

# Mail location (Maildir in user home)
mail_location = maildir:~/Maildir

# Enable auth socket for Postfix
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

For simple setups, create system users (e.g., useradd) and they become mailbox owners. For multi-domain hosting, configure virtual users with SQL; that’s beyond a beginner guide but fully supported by Postfix/Dovecot.

sudo systemctl enable --now dovecot
sudo systemctl reload dovecot

7) Spam Filtering and DKIM/DMARC

You need message signing and spam defenses to land in inboxes. Choose one path:

Option A: Rspamd (modern, fast)

Rspamd handles spam scoring and DKIM signing. Install via your distro or official repos, then configure Postfix milter to call Rspamd and create DKIM keys per domain. Basic flow:

• Install rspamd and redis (caches filters).
• Generate DKIM keys in /var/lib/rspamd/dkim/ and publish the TXT record.
• Enable milter in Postfix main.cf to pass messages through Rspamd.

Option B: SpamAssassin + OpenDKIM + OpenDMARC

This classic combo works well and is easy to reason about:

# Ubuntu
sudo apt -y install spamassassin opendkim opendkim-tools opendmarc
sudo systemctl enable --now spamassassin opendkim opendmarc

Generate DKIM keys and add the DNS record:

sudo mkdir -p /etc/opendkim/keys/example.com
cd /etc/opendkim/keys/example.com
sudo opendkim-genkey -s default -d example.com
sudo chown opendkim:opendkim default.private
# Publish default.txt as a TXT record for: default._domainkey.example.com

Wire OpenDKIM/OpenDMARC to Postfix as milters in /etc/postfix/main.cf:

milter_default_action = accept
milter_protocol = 2
smtpd_milters = inet:127.0.0.1:8891, inet:127.0.0.1:8893
non_smtpd_milters = $smtpd_milters

Reload services after configuration changes.

8) Publish SPF, DKIM, DMARC, and rDNS

Add or verify these DNS records (replace with your values):

; A and MX
mail.example.com.   IN A     203.0.113.10
example.com.        IN MX 10 mail.example.com.

; SPF
example.com. IN TXT "v=spf1 a mx ip4:203.0.113.10 ~all"

; DKIM (from your generated default.txt)
default._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqh..."

; DMARC (monitoring mode first)
_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; fo=1"

; Later tighten DMARC to p=quarantine or p=reject after testing

Confirm PTR (rDNS) maps 203.0.113.10 → mail.example.com and that it matches your SMTP banner (myhostname).

9) Create Mailboxes and Test

For system users:

sudo adduser alice
sudo -u alice maildirmake.dovecot ~/Maildir
sudo -u alice maildirmake.dovecot ~/Maildir/.Sent
sudo -u alice maildirmake.dovecot ~/Maildir/.Trash

Send a test message and check logs:

echo "Test body" | mail -s "Hello" alice@example.com
sudo tail -n 100 /var/log/mail.log    # Ubuntu/Debian
sudo journalctl -u postfix -u dovecot # RHEL/Rocky

Use an email client (Thunderbird, Outlook, Apple Mail) with IMAPS (993) and SMTP Submission (587). Ensure “Use TLS” or “STARTTLS” is on.

10) Optional: Webmail (Roundcube)

Install Roundcube and point it at localhost IMAP/SMTP with TLS. Secure the webmail vhost with HTTPS using a Let’s Encrypt certificate.

Deliverability Best Practices

• Warm up new IPs: start with a few hundred emails/day and scale gradually.
• Keep list hygiene: confirm opt-ins, remove bounces and complainers quickly.
• Set up feedback loops with major ISPs where available.
• Adopt MTA-STS and TLS-RPT TXT records to enforce and monitor TLS.
• Consider BIMI after DMARC is at p=quarantine or p=reject and you have a VMC.

Security Hardening Essentials

• Fail2ban: block abusive auth attempts against Postfix/Dovecot.
• Disable VRFY and EXPN in Postfix to prevent address harvesting.
• Enforce strong auth and unique passwords; consider rate limits per user/IP.
• Keep your server updated; auto-renew TLS certs with Certbot timers.
• Never allow open relay: test using online relay checkers.

# Example Postfix hardening fragments
smtpd_helo_required = yes
disable_vrfy_command = yes
maximal_backoff_time = 4000s
smtp_tls_mandatory_ciphers = high

Monitoring, Backups, and Ongoing Care

• Logs: watch /var/log/mail.log (Debian/Ubuntu) or journalctl on RHEL.
• Metrics: track queue length, deferred messages, and rejection reasons.
• Backups: include Maildirs, Postfix/Dovecot/Rspamd configs, and DKIM keys.
• Blacklists: check Spamhaus, SORBS, Barracuda; delist promptly with evidence.
• User education: teach strong passwords and IMAPS/Submission usage.

Common Errors and Quick Fixes

• Can’t send to Gmail/Outlook: verify rDNS, SPF, DKIM, DMARC alignment, and that port 25 is open.
• TLS errors: confirm certificate paths in Postfix/Dovecot and that the CN/SAN matches mail hostname.
• Authentication fails: ensure Dovecot auth socket permissions and matching usernames/passwords.
• Open relay warnings: check smtpd_recipient_restrictions and milter settings; must require auth.
• Messages marked spam: improve content, warm IP, enable DMARC quarantine, and reduce link-shorteners.

When to Choose Managed Email or YouStable

Running your own MTA provides control but requires vigilance. If you prefer a managed path, YouStable’s dedicated servers come with clean IPs, rDNS assistance, and 24×7 support. Our team can help you implement Postfix/Dovecot or set up a control panel and ensure SPF, DKIM, and DMARC are correctly deployed for reliable deliverability.

FAQs: Set up Email Server