Gmail on macOS command line: neomutt + offlineimap + notmuch + pass + vim + launchd (brew services) + w3m

I’ve been experimenting with a new email client on the command line. I used alpine for a long time, but I’ve heard a lot about mutt as well. It seems to be the standard terminal email program, but my past attempts to set it up have always failed, so I set aside a couple of hours over the last few days to really get into the subject. This is the setup I have so far, which uses the neomutt fork. It’s an amalgamation of lots of different guides and snippets that I’ve found scattered around on the internet. I’ve referenced the important ones at the end.

y setup is fairly particular to me and my needs. I use macOS rather than Linux which produces some interesting idiosyncrasies, and I wanted to integrate my existing command line programs where possible, notably vim (text editor) and pass (GPG enabled password manager), and w3m (web browser). I also use Gmail, so interacting with that requires understanding their slightly odd IMAP configuration.

Offlineimap

For now we won’t even install neomutt, there isn’t anything to see yet anyway and much of the config won’t make sense until we’ve set up the back end stuff. The first thing to set up is the syncing of emails with Gmail’s IMAP system.

neomutt can handle IMAP email, but grabbing from a remote server to read an email requires that there be an internet connection everytime you open neomutt. To keep an offline database of my Gmail account, so I can search it when travelling, I use offlineimap, which can also be installed using Homebrew :

brew install offlineimap

Steve Losh

offlineimap takes its configuration from ~/.offlineimaprc. This is my config:

general
ui = TTY.TTYUI
accounts = johngodlee@gmail.com 
pythonfile = ~/.offlineimap.py
fsync = False
ssl = False

Account johngodlee@gmail.com
localrepository = johngodlee-local
remoterepository = johngodlee-remote

Repository johngodlee-local
type = Maildir
localfolders = ~/.mail/johngodlee@gmail.com
nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
	'sent': '[Google Mail]/Sent Mail',
	'flagged': '[Google Mail]/starred',
	'trash': '[Google Mail]/Bin',
	}.get(folder, folder)

Repository johngodlee-remote
maxconnections = 3
type = Gmail
remoteuser = johngodlee@gmail.com
remotepasseval = get_pass()
realdelete = no
ssl=true
sslcacertfile = /usr/local/etc/openssl/cert.pem
nametrans = lambda folder: {'[Google Mail]/Drafts':    'drafts',
	'[Google Mail]/Sent Mail': 'sent',
	'[Google Mail]/Starred':   'starred',
	'[Google Mail]/Bin':     'trash',
	}.get(folder, folder)

folderfilter = lambda folder: folder not in ['[Google Mail]/Bin',
	'[Google Mail]/Important',
	'[Google Mail]/Spam',
	'[Google Mail]/Chats',
	'[Google Mail]/All Mail',
	]

There is a lot there, so here is a line by line breakdown:

ui = TTY.TTYUI signals which user interface to use. TTY.TTYUI makes sure that the process runs quietly in the background, there are also other options which may be better, but this one works for me.

accounts = johngodlee@gmail.com is the email account to be checked.

pythonfile = ~/.offlineimap.py is a python script which offlineimap calls to get the password for my email account from pass. I’ll go through .offlineimap.py later.

fsync = False just tells offlineimap that it doesn’t need to make sure a full sync is completed every time, if a few emails get lost, no bother, they will just get synced next time

Account johngodlee@gmail.com
localrepository = johngodlee-local
remoterepository = johngodlee-remote

The section above defines the names of repositories to use for this email account.

Repository johngodlee-local
type = Maildir
localfolders = ~/.mail/johngodlee@gmail.com

The section above defines the location and mailbox type of the local repository for the email from my account

nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts', ...}.get(folder, folder) translates Gmail’s IMAP folders into folders in my local mail directory. I think this is necessary as Gmail’s folders contain forward slashes, which would mess up a normal file system. I only translated the system IMAP folders, as custom folders (labels) don’t use the [Google Mail]/ prefix and are synced automatically as is. I had some trouble getting offlineimap to recognise my system folders, as many other guides recommend using the [Gmail]/ prefix, but for whatever reason my account uses [Google Mail]/.

Repository johngodlee-remote
maxconnections = 3
type = Gmail
remoteuser = johngodlee@gmail.com
remotepasseval = get_pass()
realdelete = no
ssl=true
sslcacertfile = /usr/local/etc/openssl/cert.pem

This section configures the remote repository and how to interact with it.

maxconnections = 3 defines the number of parallel connections offlineimap can make when syncing emails. 3 is low enough that Gmail won’t enforce rate limits and break the connection.

type = Gmail tells offlineimap that the account is a Gmail account, and so it should take that into account when dealing with their weird IMAP setup.

remoteuser = johngodlee@gmail.com simply tells us what account is being accessed.

remotepasseval = get_pass() tells offlineimap where to get the password for the account. get_pass() is the python function that is created in ~/.offlineimap.py

realdelete = no tells offlineimap not to totally delete an email when you press delete in neomutt, instead it will keep it in All Mail.

ssl = true says to always use ssl encryption when syncing

sslcacertfile = /usr/local/etc/openssl/cert.pem gives the location of a security certificate, which I think helps ssl to further prevent man in the middle attacks when syncing.

nametrans = lambda folder: {'[Google Mail]/Drafts': 'drafts', ...}.get(folder, folder) is just like the nametrans function earlier, only it goes in the other direction.

folderfilter = lambda folder: folder not in ['[Google Mail]/Bin', ...] gives a list of folders that should not be synced. In my case, I don’t want to sync the trash (Bin because British), Spam, Chats, and All Mail.

Later on I’ll look at how to get offlineimap to run in the background using launchd, a macOS alternative to crontab.

Pass and .offlineimap.py

#!/usr/bin/env python

from subprocess import check_output

def get_pass():
  return check_output("/usr/local/bin/pass email/johngodlee@gmail.com", shell=True).splitlines()[0]

This script looks in pass for the entry email/johngodlee@gmail.com, which contains the password for my email account, and then takes the first line (splitlines()[0]) of that entry to store it in get_pass(). This function is then called by remotepasseval in .offlineimaprc. The script should be saved as ~/.offlineimap.py.

Other programs

notmuch

notmuch provides very fast email searching. It has other capabilities such as tagging and indexing email, but I use Gmail’s IMAP folders (labels) for this, so I only use it for searching. notmuch can be installed via Homebrew:

brew install notmuch

When installed, neomutt allows searching using notmuch by defining a custom macro. You can then use all the notmuch search syntax to quickly find emails within neomutt. I’ll go through that in the next section where I define the muttrc.

w3m

Neomutt doesn’t know how to display emails that are encoded as HTML only. To render HTML as something legible in the terminal I use w3m. This is a text based web browser. Once again, this can be installed using Homebrew:

brew install w3m

To tell neomutt to forward HTML emails to w3m for plain text encoding, then send it back to neomutt, we need to create a file called .mailcap. Keep it in the home directory as convention:

touch ~/.mailcap

Ten add the following snippet to the file:

text/html; w3m -dump -o document_charset=%{charset} '%s'; nametemplate=%s.html; copiousoutput

.mailcap is then referenced later in the muttrc.

Neomutt

Now that we’ve set up all the background, we can install the central part of the workflow, mutt. Specifically, I’m using neomutt, which is a fork of the original mutt which incorporates some of the most widely used plugins.

neomutt can be installed using Homebrew:

brew install neomutt

Then we need to make a configuration file, I like to keep mine in ~/.mutt/muttrc but see man neomutt for more locations where neomutt will look for a config file:

mkdir ~/.mutt

touch ~/.mutt/muttrc

The first thing to add to the configuration is some IMAP settings:

set imap_user = "johngodlee@gmail.com"

## Call pass from within 
set my_pass = "`pass email/johngodlee@gmail.com`" 
set imap_pass = $my_pass

set from = "johngodlee@gmail.com"
set realname = "John Godlee"

set folder = "~/.mail/johngodlee@gmail.com"
set spoolfile = +INBOX
set postponed = +drafts
set record = +sent

## Don't automatically move messages after reading
set move = no

## Max time mutt should wait before polling IMAP connections, in minutes
set imap_keepalive = 30  

## Set smtp URL for replying 
set smtp_url = "smtp://johngodlee@smtp.gmail.com:587/"

## Set the login method for smtp replying
set smtp_pass = $my_pass
set smtp_authenticators = 'login'

## Allow mutt to open new imap connections to test for new mail 
unset imap_passive

## Set cache locations so IMAP polling is quicker on startup
set header_cache = ~/.mutt/johngodlee/headers
set message_cachedir = ~/.mutt/johngodlee/bodies
set certificate_file = ~/.mutt/certificates

## If stuck in a prompt, abort after n minutes to check IMAP (timeout), n minutes between checks (mail_check)
set timeout = 10
set mail_check = 5

Some of this is self explanatory, so I’ll only go through the important bits

set my_pass ... calls pass to create the variable my_pass which is then sent to imap_pass to set the password for my Gmail account, which is needed for sending mail.

set folder ... sets the location of the mailbox which offlineimap syncs email to. The value of folder also acts as the prefix for set spoolfile, set postponed and set record, as indicated by putting + before the folder names.

set spoolfile ... should be the location of your inbox, where new mail arrives

set postponed ... is where unfinished new emails are kept if they are aborted before being sent

set record ... should be a folder where a copy of sent messages are kept

Note that set smtp_pass ... also calls the my_pass variable

As neomutt comes with the sidebar patch, I wanted to make use of it by displaying any folders that are synced from offlineimap:

## Mailboxes to display in sidebar and check regularly 
mailboxes +INBOX +'starred' +'trash' +'drafts' +'sent' +'Archived' +'coding_club' +'diss_manuscript' +'personal' +'PhD' +'SEOSAW_contacts' +'SEOSAW_logos' +'STEB_2018' +'urgent' +'hemi_lens_proposal' +kew_taxonomy_course +'angola_botanic_garden'

## Sort sidebar folders alphabetically
set sidebar_sort_method = name

## Sidebar visible by default
set sidebar_visible = yes

## Don't abbreviate folders in sidebar
set sidebar_short_path = no

I use a stripped down .vimrc for writing email, as I don’t need a lot of the plugins that I use for programming. I can tell neomutt to write emails using vim with this alternative .vimrc (.vimrc_alpine) by adding the following to the neomuttrc:

set editor = 'vim -u ~/.vimrc_alpine'

Here is my .vimrc_alpine:

set nocompatible              " be iMproved, required
filetype off                  " required

" enable syntax highlighting
syntax on

" Stop creating swp and ~ files
set nobackup
set noswapfile

" Ignore case of searches
set ignorecase

" Don’t reset cursor to start of line when moving around
set nostartofline

" Preserve indentation on wrapped lines
set breakindent

" Disable folding in markdown
let g:vim_markdown_folding_disabled = 1

" Disable syntax conceal in markdown
let g:vim_markdown_conceal = 0

" Normal backspace behaviour
set backspace=2

" map A (append at end of line) to a (append in place)
nnoremap a A

" Move by visual lines rather than actual lines with k,j
nnoremap k gk
nnoremap j gj
nnoremap gk k
nnoremap gj j

" Easier save and quit with ;
noremap ;w :w<CR>
noremap ;q :q<CR>

" Copy and paste from `+` register for interacting with mac clipboard
vnoremap y "+y 
vnoremap p "+p
nnoremap p "+gp
vnoremap d "+d
nnoremap dd "+dd

" Map of modes and their codes for statusline
let g:currentmode={
    \ 'n'  : 'N ',
    \ 'no' : 'N·Operator Pending ',
    \ 'v'  : 'V ',
    \ 'V'  : 'V·Line ',
    \ '^V' : 'V·Block ',
    \ 's'  : 'Select ',
    \ 'S'  : 'S·Line ',
    \ '^S' : 'S·Block ',
    \ 'i'  : 'I ',
    \ 'R'  : 'R ',
    \ 'Rv' : 'V·Replace ',
    \ 'c'  : 'Command ',
    \ 'cv' : 'Vim Ex ',
    \ 'ce' : 'Ex ',
    \ 'r'  : 'Prompt ',
    \ 'rm' : 'More ',
    \ 'r?' : 'Confirm ',
    \ '!'  : 'Shell ',
    \ 't'  : 'Terminal '
    \}

" Change statusline based on colour
function! ChangeStatuslineColor()
  if (mode() =~# '\v(n|no)')
    exe 'hi! StatusLine ctermfg=112'
  elseif (mode() =~# '\v(v|V)' || g:currentmode[mode()] ==# 'V·Block' || get(g:currentmode, mode(), '') ==# 't')
    exe 'hi! StatusLine ctermfg=172'
  elseif (mode() ==# 'i')
    exe 'hi! StatusLine ctermfg=044'
  else
    exe 'hi! StatusLine ctermfg=007'
  endif
  return ''
endfunction

" Make statusline always show
set laststatus=2

" statusline
	" left side
	set statusline=%{ChangeStatuslineColor()}	" Change colour
	set statusline+=%0*\ %{toupper(g:currentmode[mode()])} 	" Current mode
	set statusline+=\
	set statusline+=%1*%m%r " Modified and read only flags
	set statusline+=%1*%=	" Switch to right side

	" right side
	set statusline+=%0*\ 		"Space
	set statusline+=%0*%y	" File syntax
	set statusline+=%0*\|	" Vert-line
	set statusline+=%0*%p%%	" Percentage through file
	set statusline+=%0*\|\ 	" Vert-line and Space
	set statusline+=%0*%l:%c	" Line and column number
	set statusline+=%0*\ 		" Space

" Set colours for statusline middle section
hi User1 ctermfg=255 ctermbg=240

" Ragged right line breaks
set linebreak

I want to read emails in plain text if possible, with the provided plain text being used preferentially, and a converted HTML version if plain text isn’t available, which I can set using alternative_order .... I also want to tell neomutt how to deal with forced HTML emails, which is to automatically view them and also to give the path to the .mailcap file we created earlier, which uses w3m to parse HTML as plain text:

# Read in plain text if possible
alternative_order text/plain text/html
auto_view text/html
set mailcap_path = ~/.mailcap

# Enforce encoding in utf8
set send_charset="utf-8"

To allow notmuch to work I need to allow neomutt to create virtual folders when searching and also to give the location of the mailbox to search (set nm_default_uri ...)

I also set up a keybinding to initiate the notmuch search from within neomutt as \\

set virtual_spoolfile = yes   
set nm_default_uri = "notmuch:///Users/johngodlee/.mail/johngodlee@gmail.com"
macro index,pager \\  "<vfolder-from-query>"  

Launchd

Other guides that I read recommended that I use crontab to schedule offlineimap updates. But on macOS launchd is the proper way to do things. macOS uses .plist scripts stored in ~/Library/LaunchAgents to designate jobs, which use an XML format. Helpfully, Homebrew services can generate .plist scripts for many programs, which can then be amended later.

To start running Homebrew services, first see what services are listed:

brew services list

offlineimap should be listed in the output.

To generate the script to start the service at login:

brew services start offlineimap

To ensure that notmuch follows behind offlineimap and updates its database after every pull from Gmail, I added some extra commands to the .plist file which Homebrew services generated:

vim ~/Library/LaunchAgents/homebrew.mxcl.offlineimap.plist

Then amend the <array> section so it looks like this:

    <array>
      <string>/usr/local/opt/offlineimap/bin/offlineimap</string>
      <string>-u</string>
      <string>quiet</string>
      <string>;</string>
      <string>notmuch</string>
      <string>new</string>
    </array>

This runs a notmuch new everytime offlineimap runs. The other flags -u and quiet tell offlineimap not to print anything to the terminal which could disrupt the job running in the background.

Guides I used

With all this setup you should be able to run offlineimap, then open neomutt and start reading and writing email, though it’s entirely possible that I may have missed something. In any case, here are the guides that I learnt from and used to write this setup:

• https://wiki.archlinux.org/index.php/mutt
• http://stevelosh.com/blog/2012/10/the-homely-mutt/
• https://smalldata.tech/blog/2016/09/10/gmail-with-mutt
• https://gist.github.com/amandabee/cf7faad0a6f2afc485ee
• https://github.com/cbracken/mutt
• https://www.farces.com/wikis/naked-server/homebrew/brew-services/
• https://bbs.archlinux.org/viewtopic.php?id=142377
• https://pbrisbin.com/posts/mutt_gmail_offlineimap/
• https://baptiste-wicht.com/posts/2014/07/a-mutt-journey-my-mutt-configuration.html

y full setup can be explored in my dotfiles