“smart”quotes in Outlook on Mac

Spent ages trying to figure out how to stop Outlook 2011 converting my straight “quotes’ into “smart”(curley, UTF-8, doesn’t render well on Windows machines)-quotes.

Turns out it’s not an Outlook setting at all but a system-wide OSX setting in the Keyboard preferences which controls how all text-input boxes work:

Screen Shot 2015-02-16 at 10.34.36

Quick and dirty delete all top-level files in a directory not owned by any user with a process currently running

Handy for shared systems where you want to reap files in, for example, /tmp without affect currently running processes. There are some unhandled exceptions, such as files disappearing after the file list is built but before the file is deleted (which will throw an uncaught file not found exception) but as a quick and dirty first attempt I think it’s not too shabby.

#!/usr/bin/env python

import collections
import logging
import getopt
import os
import shutil
import sys

def usage():
        sys.stderr.write("Usage: {scriptname} [[-h | -? | --help] | [-d | --debug] path [path1 [path2...]]]\n".format(scriptname=sys.argv[0]))

def remove(path):
        if os.path.isdir(path):
                shutil.rmtree(path)
        else:
                os.remove(path)

def reap(directory):

        # Get a list of all files in the directory to consider for reaping, and group them by owner's uid
        user_files = collections.defaultdict(list)
        map(lambda file: user_files[os.lstat(os.path.join(directory, file)).st_uid].append(file), os.listdir(directory))

        # Get a list of users who have processes running on the box
        users_with_processes = [ os.lstat('/proc/{proc}'.format(proc=proc)).st_uid for proc in os.listdir('/proc') if proc.isdigit() ]

        # Now find the users who do not have running processes, as these are the users' whose files we are going to reap (always skip root)
        users_to_reap = [ user for user in user_files.keys() if user != 0 and user not in users_with_processes ]

        # remove the files
        if DEBUG:
                action=logging.debug
        else:
                action=remove
        map(action, [ os.path.join(directory, file) for file in [ file for user in users_to_reap for file in user_files[user] ] ])

try:
        optlist, args = getopt.getopt(sys.argv[1:], 'dh?', ['debug', 'help'])
except getopt.GetoptError as err:
        sys.stderr.write(str(err) + "\n")
        usage()
        sys.exit(2)

DEBUG=False

for opt, value in optlist:
        if opt in ('-d', '--debug'):
                DEBUG=True
        elif opt in ('-h', '-?', '--help'):
                usage()
                sys.exit()
        else:
                sys.stderr.write("Unhandled option: {opt}\n".format(opt=opt))
                usage()
                sys.exit(2)

if len(args) == 0:
        usage()
        sys.exit(1)

if DEBUG:
        logging.basicConfig(level=logging.DEBUG)

for dir in args:
        reap(dir)

Preventing git commits as root

This is a quick pre-commit git hook to prevent committing annonymously as root – it refuses to allow root to commit directly and insists that –author is given if a user is commmitting via sudo.


#!/bin/bash

abort=0

# Check that if committing as root the author is set, as far as possible.
if [ "$UID" -eq 0 ]
then
echo "Warning: Committing as root." >&2
if [ -n "$SUDO_USER" ]
then
if ! echo $SUDO_COMMAND | grep -q -e '--author'
then
cat - >&2 < or (if you have committed before - see 'man git-commit'):
git commit --author=some_pattern_that_matches_your_name

Previous authors in repository:
$( git log --all --format='%an <%ae>' | sort -u )

EOF
abort=1
fi
else
echo "Committing as root, without using sudo. Please do not do this." >&2
abort=1
fi
fi

if [ "$abort" -ne 0 ]
then
echo -e "\n\ncommit aborted\n" >&2
exit 1
fi

Turning off GMails draconian spam filter

Okay, so there is no way to turn off GMails spam filter (or even turn it down to the point where it stops putting more legitimate emails than spam into the “Spam” folder).

To fix this behavior I have thrown together this short python script that simply moves any email found in the Spam folder into the Inbox. I used to achieve the same thing by creating a filter which told GMail not to spam any email that has an ‘@’ in the ‘From’ address but GMail has suddenly decided to start ignoring that filter so a more permanent solution is required.

#!/usr/bin/env python

import imaplib

IMAP_USER='<your_user_name>@gmail.com'
IMAP_PASSWORD='<your_password>'


if __name__ == '__main__':
  imap4 = imaplib.IMAP4_SSL('imap.gmail.com')
  imap4.login(IMAP_USER, IMAP_PASSWORD)
  imap4.select('[Gmail]/Spam')
  typ, data = imap4.search(None, 'ALL')
  for num in data[0].split():
    message_subj = imap4.fetch(num, '(BODY.PEEK[HEADER.FIELDS (SUBJECT FROM TO DATE)])')[1]
    print "Moving message '%s' from Spam to INBOX" % (', '.join(message_subj[0][1].rstrip().split("\r\n")))
    imap4.copy(num, 'INBOX')
    imap4.store(num, '+FLAGS', '\\Deleted')
  imap4.expunge()
  imap4.close()
  imap4.logout()

Automated notification of uncommitted git-controlled config changes

Yesterday I wrote about version controlling server configuration with GIT.

Inevitably I will change a file under version control but forget that it is under version control. To mitigate this I’ve thrown together this simple bash script:

#!/bin/bash

CONFIG_REPO="/root/vc"
MAILTO="spam@dev.null"

GIT_CMD="git status --porcelain"

tempfile=`mktemp`

cd "$CONFIG_REPO"
$GIT_CMD > $tempfile

if [ -s $tempfile ]
then
sendmail -oi -t <<EOF
To: $MAILTO
From: $USER@`hostname -f`
Subject: Uncommitted config files detected on `hostname`

I have detected that 'git' believes there are modified and/or added files which have not been committed on `hostname` in '$CONFIG_REPO'.

Hostname: `hostname -f`
Uname: `uname -a`
Repository location: $CONFIG_REPO

Output of '$GIT_CMD':
-----8<-------------------->8-----
`cat $tempfile`
-----8<-------------------->8-----

EOF
fi

rm $tempfile

I’ve put the script in /usr/local/sbin/check-config-git and scheduled it to run daily at midnight via cron. The idea is that if it annoys me every day, as opposed to once a week or even less frequently, I might actually be motivated into doing something about it.

Version controlling server configuration with GIT

Often I want to version control certain, usually critical, system configuration files. In the past I’ve either set this up on a directory-by-directory basis or not bothered (which results in me creating a lot of superfluous files by doing a ‘cp config_file config_file.bak_`date +%F`’ before editing).

I, with a colleague, have come up with this solution that, while not necessarily perfect, is a lot more manageable then previous alternatives I have used and has the benefits of creating a centralised repository on a given machine as well as not polluting system directories with ‘.svn’ or ‘.git’ directories. It also avoids having to play with nested repositories (i.e. directories with a ‘.git’ dir under another that also has a ‘.git’ dir), which are unlikely if you’re just version controlling /etc but more common if controlling /home/$USER. I think it’s quite neat and keeps the revision control itself away from the core system files.

  1. The first step is to create a suitable directory to host the version control. We won’t actually be storing files here but it does need to be a “normal” repository (i.e. not a bare repository) as it does represent a working copy of the repository.

    mkdir /root/vc # Calling it 'vc' for 'Version Control'
    cd /root/vc

  2. Step two is to initialise our repository:

    git init

  3. Now, and this is the clever bit, we need to configure the repository to use ‘/’ as the base of it’s working tree (so we end up with a repository, with all it’s revision control files, in /root/vc but really the files under ‘/’ is under revision control). I also excluded everything by default (so git does not list everything as not being controlled and we can cherry pick the files we actually care about). UPDATE: I’ve since found the config variable ‘staus.showUntrackedFiles’ (reading man-pages FTW!), which achieves the same end in a much saner manner.

    git config core.worktree /
    echo '*' >> .git/info/exclude
    git config status.showUntrackedFiles no

…and that’s it. Just use the ‘/root/vc’ directory as a normal git repository but with any files under ‘/’. Simple, eh?

There is a drawback, however. Since I have excluded everything, files have to be added to the repository with a ‘-f’ (force) flag:

git add -f /etc/ssh/sshd_config

This also applies when using ‘git add’ to stage modified files however ‘git commit -a’, which stages modified & deleted files and commits in one step, does not required ‘-f’.
UPDATE: This is no longer an issue using ‘status.showUntrackedFiles’ to disable showing untracked files by default. There maybe other issues with this approach but I’ve not spotted them in my (5 minutes!) of testing/experimentation.

There should be more than one way to do it in Python

Python has a philosophy of ‘There should be one– and preferably only one –obvious way to do it’ (http://www.python.org/dev/peps/pep-0020/) rather then Perl’s ‘There’s more than one way to do it’ (Programming Perl Third Edition, Larry Wall et al.). This is great, in theory – it leads to greater consistency between disparate programs and makes it easier for individual programmers to pick up someone else’s code.

The problem comes where the obvious way is not, for whatever reason, the practical way. For example, the obvious way to test if a string begins with another string is to use "string1".startswith("string2"). This is, however, significantly less performant than doing "string1"[7:] == "string2" which means when doing a large number of these tests you have to use this form, despite it not being the obvious method.

Unless you are familiar with Python’s sequence slicing syntax (http://docs.python.org/library/stdtypes.html#typesseq) I do not find "string1"[7:] to be obvious (even though I would expect most programmers to be able to hazard a, probably correct, guess to what it does), which means I would precede that line with a # Check if string1 starts with "string2". When I feel the need to comment on a specific line of code it is usually because I do not think what it is doing is sufficiently obvious, which means it violates Python’s ‘There should be one– and preferably only one –obvious way to do it’.

Just a quickie…

Spent about 20 minutes trying to figure out why this line in fstab allowed me to mount, but not unmount, a cifs share as a “normal” user:

//isolinear/documents /media/isolinear cifs user=laurence,file_mode=0644,dir_mode=0755,noauto,user 0 0

The error I was getting was:

umount: /media/isolinear mount disagrees with the fstab

It turns out that mount.cifs adds an extra ‘/’ to the end of the source mountpoint (i.e. ‘//isolinear/documents’ becomes ‘//isolinear/documents/’) causing the mismatch. Adding the extra ‘/’ in fstab meant I could mount and unmount it happily as a non-root user.

Posted in the hope that someone else might happen upon this entry when experiencing the same issue!

Scientific Linux 6 (and, by extension, RHEL6) authentication woes

Since no-one seems willing to fork out the cash for RHEL, yet management insist on using CentOS as “our skills are with RedHat”, and there is no sign yet of CentOS6 I have been doing some experimenting with Scientific Linux 6 (which is also a RHEL rebuild).

Configuring the new sssd daemon to do AD authentication seemed straightforward enough until I hit an interesting problem. It would appear that, beginning with RHEL6, RedHat has split /etc/pam.d/system-auth into /etc/pam.d/system-auth and /etc/pam.d/password-auth. In of itself this is not a problem HOWEVER I have also discovered that, out of the box, GDM uses the /etc/pam.d/gdm-password stack (which includes password-auth) and gnome-screensaver uses the /etc/pam.d/gnome-screensaver stack (which includes system-auth). The result of this is that if I configure system-auth only (which is what the RHEL6 deployment guide says to do[0] then I cannot login to GDM. If I set up password-auth only (described in the migration guide[1] as for “remote services”) I can login to GDM but once my session is locked with gnome-screensaver (either manually or by the screensaver timeout (which is on and locks by default) I cannot unlock it from the prompt which appears when I click the mouse or touch a key (although I can click “switch user” to get back to GDM where I can unlock it). Setting both up seems counter-intuitive if I only want to configure local access and ‘password-auth’ is for remote services.

[0] http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/chap-SSSD_User_Guide-Setting_Up_SSSD.html
[1] http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Migration_Planning_Guide/ch07s05.html

even more Python vs Perl performance

Following my last two posts (http://blog.entek.org.uk/?p=106 and http://blog.entek.org.uk/?p=112) I took some profilers to my codes.

First up my revised Python implementation:

> python -m cProfile checkmail2
...
         16225093 function calls in 8.663 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    8.663    8.663 :1()
        1    0.000    0.000    0.000    0.000 UserDict.py:17(__getitem__)
        1    5.906    5.906    8.662    8.662 checkmail2:3()
        1    0.000    0.000    8.663    8.663 {execfile}
       37    0.000    0.000    0.000    0.000 {method '__enter__' of 'file' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
       22    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
 16224990    2.755    0.000    2.755    0.000 {method 'startswith' of 'str' objects}
       37    0.001    0.000    0.001    0.000 {open}
        1    0.000    0.000    0.000    0.000 {posix.listdir}

Quite clearly there is a huge number (16 million!) of calls to startswith which is the biggest time-sink outside the main script.

Comparing the Perl implementation:

> perl -d:DProf checkmail3
...
> dprofpp
Total Elapsed Time = 3.426896 Seconds
  User+System Time = 3.401494 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 0.12   0.004  0.007      4   0.0010 0.0016  main::BEGIN
 0.03   0.001  0.001      5   0.0002 0.0002  File::Basename::BEGIN
 0.03   0.001  0.001      1   0.0009 0.0009  warnings::BEGIN
 0.03   0.001  0.001     37   0.0000 0.0000  File::Basename::_strip_trailing_sep
 0.03   0.001  0.001     37   0.0000 0.0000  File::Basename::fileparse
 0.00   0.000  0.002     37   0.0000 0.0000  File::Basename::basename
 0.00   0.000  0.000      1   0.0003 0.0003  File::Glob::doglob
 0.00   0.000  0.000      1   0.0001 0.0001  DynaLoader::dl_load_file
 0.00   0.000  0.000      1   0.0001 0.0003  XSLoader::load
 0.00   0.000  0.000      1   0.0001 0.0001  File::Basename::fileparse_set_fstype
 0.00   0.000  0.000      1   0.0001 0.0001  Exporter::import
 0.00   0.000  0.000      2   0.0000 0.0000  warnings::import
 0.00   0.000  0.000      3   0.0000 0.0000  strict::import
 0.00   0.000  0.000      1   0.0000 0.0003  File::Glob::csh_glob
 0.00   0.000  0.000      1   0.0000 0.0000  strict::bits

Ignoring the actual times which are not directly comparable due to the profiling overheads we can clearly see Perl is benefiting hugely from the inbuilt regex engine as there are 0 function calls associated with each line check.

I did replace the ‘str.startswith’ implementation of the Python script with a version which used ‘re’ regex objects, but this showed even worse performance:

> time python checkmail4
...
python checkmail4 8.42s user 0.33s system 99% cpu 8.765 total

Profiling this one we see the overhead of using ‘re.match’ was about double that of ‘str.startswith’ and, obviously, the number of calls remained the same. On top of this I introduced additional overhead of two calls to ‘re.compile’ at the start of the script, which the profiler showed incurred not insignificant function calls of their own:

> python -m cProfile checkmail4
...
         16225312 function calls in 12.416 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   12.416   12.416 :1()
        1    0.000    0.000    0.000    0.000 UserDict.py:17(__getitem__)
        1    6.662    6.662   12.415   12.415 checkmail4:3()
        2    0.000    0.000    0.000    0.000 re.py:186(compile)
        2    0.000    0.000    0.000    0.000 re.py:227(_compile)
        2    0.000    0.000    0.000    0.000 sre_compile.py:367(_compile_info)
        2    0.000    0.000    0.000    0.000 sre_compile.py:38(_compile)
        4    0.000    0.000    0.000    0.000 sre_compile.py:480(isstring)
        2    0.000    0.000    0.000    0.000 sre_compile.py:486(_code)
        2    0.000    0.000    0.000    0.000 sre_compile.py:501(compile)
       15    0.000    0.000    0.000    0.000 sre_parse.py:144(append)
        2    0.000    0.000    0.000    0.000 sre_parse.py:146(getwidth)
        2    0.000    0.000    0.000    0.000 sre_parse.py:184(__init__)
       21    0.000    0.000    0.000    0.000 sre_parse.py:188(__next)
        2    0.000    0.000    0.000    0.000 sre_parse.py:201(match)
       19    0.000    0.000    0.000    0.000 sre_parse.py:207(get)
        2    0.000    0.000    0.000    0.000 sre_parse.py:307(_parse_sub)
        2    0.000    0.000    0.000    0.000 sre_parse.py:385(_parse)
        2    0.000    0.000    0.000    0.000 sre_parse.py:669(parse)
        2    0.000    0.000    0.000    0.000 sre_parse.py:73(__init__)
        2    0.000    0.000    0.000    0.000 sre_parse.py:96(__init__)
        2    0.000    0.000    0.000    0.000 {_sre.compile}
 16224990    5.751    0.000    5.751    0.000 {built-in method match}
        1    0.000    0.000   12.416   12.416 {execfile}
        6    0.000    0.000    0.000    0.000 {isinstance}
       44    0.000    0.000    0.000    0.000 {len}
       37    0.000    0.000    0.000    0.000 {method '__enter__' of 'file' objects}
       59    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
       24    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        2    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {min}
       37    0.002    0.000    0.002    0.000 {open}
       13    0.000    0.000    0.000    0.000 {ord}
        1    0.000    0.000    0.000    0.000 {posix.listdir}

Quite clearly from the profiler output each call to either ‘str.startswith’ or ‘re.match’ use a very small amount of processor time (too small to be output) but the cumulative effect of 16 million calls is where the big slowdown was occurring. To get around this I tried implementing the ‘str.startswith’ version using string splicing (i.e. “line[:5] == ‘From ‘” rather than “line.startswith(‘From ‘)”) and the result was dramatic:

> time python checkmail4
...
python checkmail4 3.86s user 0.31s system 99% cpu 4.186 total

The profiler output for this version shows that the number of function calls is now on a par with the Perl implementation:

> python -m cProfile checkmail4
...
         110 function calls in 4.311 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    4.311    4.311 :1()
        1    0.000    0.000    0.000    0.000 UserDict.py:17(__getitem__)
        1    4.308    4.308    4.311    4.311 checkmail4:3()
        1    0.000    0.000    4.311    4.311 {execfile}
       37    0.000    0.000    0.000    0.000 {method '__enter__' of 'file' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
       29    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
       37    0.003    0.000    0.003    0.000 {open}
        1    0.000    0.000    0.000    0.000 {posix.listdir}

This puts the Python version within 0.6s of the Perl version, which is close enough for me. Especially considering this is effectively comparing Perl to Python on Perl’s hometurf of text matching.

I think Perl would probably still out perform Python if I was wanting to do something more fancy involving regex substitutions but Python’s performance issues, in this case, seem to be purely down to function call overheads which Perl sidesteps by incorporating the regex engine into the core language.