This all started with a request from one of our users who had a very specific issue. Their workflow makes heavy use of the cd ~username syntax, and for difficult to spell usernames, they relied on tab completion which broke when their machine was updated.

More experienced admins than myself will probably be able to guess the issue from the beginning, especially once they know that we moved to RHEL7 with SSS. However, I am fond of the journey that this issue took me on which is why I’ve chosen to write about it.

So, where do bash autocompletions actually come from? And why did this one in particular break when we switched over to RHEL 7?

Other than installing the bash-completions package and just having completions automatically work I had never used programmatic completion before. After some quick searching I found that bash completions are made possible with the complete built-in function.

From the manual we can get a printout of existing completions by running complete without any arguments. Here’s what I get when I run it on my machine.

complete | grep cd
complete -o nospace -F _cd pushd
complete -F _filedir_xspec cdiff
complete -o nospace -F _cd cd

From this it seems that _cd is the function we’re looking for. Some research reveals that it’s customary to register the completion function for a command foo as _foo.

So what does the _cd function do? We can have bash print the body of a function with the type command.

type _cd
_cd is a function
_cd ()
{
    local cur prev words cword;
    _init_completion || return;
    local IFS='
' i j k;
    compopt -o filenames;
    if [[ -z "${CDPATH:-}" || "$cur" == ?(.)?(.)/* ]]; then
        _filedir -d;
        return;
    fi;
    local -r mark_dirs=$(_rl_enabled mark-directories && echo y);
    local -r mark_symdirs=$(_rl_enabled mark-symlinked-directories && echo y);
    for i in ${CDPATH//:/'
'};
    do
        k="${#COMPREPLY[@]}";
        for j in $( compgen -d -- $i/$cur );
        do
            if [[ ( -n $mark_symdirs && -h $j || -n $mark_dirs && ! -h $j ) && ! -d ${j#$i/} ]]; then
                j+="/";
            fi;
            COMPREPLY[k++]=${j#$i/};
        done;
    done;
    _filedir -d;
    if [[ ${#COMPREPLY[@]} -eq 1 ]]; then
        i=${COMPREPLY[0]};
        if [[ "$i" == "$cur" && $i != "*/" ]]; then
            COMPREPLY[0]="${i}/";
        fi;
    fi;
    return
}

And people say shell code isn’t pretty? Anyway, the takeaway is that _cd is basically a wrapper around the compgen command that does all the heavy lifting. The next step will be investigating how this command works.

Straight to The Source⌗

Since compgen is a bash built-in if we want it to divulge its secrets we’ll have to inspect the source. Doing our due diligence would require that we use the source for the exact same version of bash that we’re using, including all of RedHat’s patches, but since we’re just poking around we can probably use the latest version safely.Since compgen is a bash built-in if we want it to divulge its secrets we’ll have to inspect the source. Doing our due diligence would require that we use the source for the exact same version of bash that we’re using, including all of RedHat’s patches, but since we’re just poking around we can probably use the latest version safely.

Looking around the directories the most relevant file for our purposes seems to be builtins/complete.def which contains the source for compgen. Here’s what we find in the compgen_builtin function.

cs->funcname = STRDUP (Farg);
  cs->command = STRDUP (Carg);
  cs->filterpat = STRDUP (Xarg);

  rval = EXECUTION_FAILURE;
  sl = gen_compspec_completions (cs, "compgen", word, 0, 0, 0);

  /* If the compspec wants the bash default completions, temporarily
     turn off programmable completion and call the bash completion code. */
  if ((sl == 0 || sl->list_len == 0) && (copts & COPT_BASHDEFAULT))
    {
      matches = bash_default_completion (word, 0, 0, 0, 0);
      sl = completions_to_stringlist (matches);
      strvec_dispose (matches);
    }

From this it seems that, at lest when bash isn’t performing the default completion, that it offloads the actual work to the function gen_compspec_completions. After searching around the source tree we discover that this function lives in the file pcomplete.c. Here is the relevant excerpt from its definition.

#ifdef DEBUG
  debug_printf ("gen_compspec_completions (%s, %s, %d, %d)", cmd, word, start, end);
  debug_printf ("gen_compspec_completions: %s -> %p", cmd, cs);
#endif
  ret = gen_action_completions (cs, word);
#ifdef DEBUG
  if (ret && progcomp_debug)
    {
      debug_printf ("gen_action_completions (%p, %s) -->", cs, word);
      strlist_print (ret, "\t");
      rl_on_new_line ();
    }
#endif

In the same file we see that gen_action_completions has the following in its definition.

GEN_XCOMPS(flags, CA_COMMAND, text, command_word_completion_function, cmatches, ret, tmatches);
GEN_XCOMPS(flags, CA_FILE, text, pcomp_filename_completion_function, cmatches, ret, tmatches);
GEN_XCOMPS(flags, CA_USER, text, rl_username_completion_function, cmatches, ret, tmatches);
GEN_XCOMPS(flags, CA_GROUP, text, bash_groupname_completion_function, cmatches, ret, tmatches);
GEN_XCOMPS(flags, CA_SERVICE, text, bash_servicename_completion_function, cmatches, ret, tmatches);

It’s not important what this macro actually does because all we need is the fact that it’s using the function rl_username_completion_function from the GNU readline package. Finally! A light at the end of the tunnel!

Readline? Isn’t that an input library?⌗

Not knowing anything about readline, other than that bash uses it to process user input, I wouldn’t have guessed that it had built-in completion capabilities. Luckily our copy of bash comes with lib/readline so we don’t have to download it separately.

In the file lib/readline/complete.c we find the following definition of our function rl_username_completion_function.

char *
rl_username_completion_function (text, state)
     const char *text;
     int state;
{
#if defined (__WIN32__) || defined (__OPENNT)
  return (char *)NULL;
#else /* !__WIN32__ && !__OPENNT) */
  static char *username = (char *)NULL;
  static struct passwd *entry;
  static int namelen, first_char, first_char_loc;
  char *value;

  if (state == 0)
    {
      FREE (username);

      first_char = *text;
      first_char_loc = first_char == '~';

      username = savestring (&text[first_char_loc]);
      namelen = strlen (username);
#if defined (HAVE_GETPWENT)
      setpwent ();
#endif
    }

#if defined (HAVE_GETPWENT)
  while (entry = getpwent ())
    {
      /* Null usernames should result in all users as possible completions. */
      if (namelen == 0 || (STREQN (username, entry->pw_name, namelen)))
        break;
    }
#endif

  if (entry == 0)
    {
#if defined (HAVE_GETPWENT)
      endpwent ();
#endif
      return ((char *)NULL);
    }
  else
    {
      value = (char *)xmalloc (2 + strlen (entry->pw_name));

      *value = *text;

      strcpy (value + first_char_loc, entry->pw_name);

      if (first_char == '~')
        rl_filename_completion_desired = 1;

      return (value);
    }
#endif /* !__WIN32__ && !__OPENNT */
}

Hallelujah, we finally found it. All this to just find out that it’s calling the libc function getpwent. This is where those experienced admins will probably sigh and utter something like “well obviously” but hey, at least now we have proven it.

The Upshot⌗

How does this knowledge help us with our problem? Well we know with certainty that username predictions originate from a call to getpwent so we should look at what is returned by the commandl getent passwd.

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/usr/bin/nologin
daemon:x:2:2:daemon:/:/usr/bin/nologin
mail:x:8:12:mail:/var/spool/mail:/usr/bin/nologin
ftp:x:14:11:ftp:/srv/ftp:/usr/bin/nologin
http:x:33:33:http:/srv/http:/usr/bin/nologin
...

Only local users! It’s no wonder our tab completion isn’t finding our users because they’re simply not in the database. And from where is this database populated? To find out we need to look in /etc/nsswitch.conf.

passwd:     files sss

Now we’ve really narrowed down the problem, SSS isn’t returning any of our domain users. Can we pull an individual user’s information?

$ getent passwd $me
me:x:$uid:$gid:$me:$homedir:/bin/bash

So we can query individual users, which would explain how we were able to log in in the first place, but for some reason we can’t list the information of every user. Isn’t there a SSS setting about that? In sssd.conf(5) we find just what we’re looking for.

enumerate (bool)
    Determines if a domain can be enumerated. This parameter can have one of the following values:

    TRUE = Users and groups are enumerated

    FALSE = No enumerations for this domain

    Default: FALSE

After enabling this setting and restarting sssd we can now see all of the domain users in the passwd database and completions start working again!

Conclusion⌗

To me this is one of the big advantages of open source. If you don’t understand how something works, the source will happily divulge. Sure, it’s not something you’ll do every day, but it certainly comes in handy when you need it.