Trace Data

Ramblings on devops, data, games, programming, and more

Chef, vBulletin, Iconv, and LC_ALL

| Comments

Skip to the End…

If the LC_ALL environment variable is not set, Chef will use ‘C’ as the default value which may cause headaches.

For now, I recommend either setting the environment variable to the empty string in a cookbook:

1
    ENV['LC_ALL']=''

or setting it in an execute block for a command that may have an issue:

1
2
3
4
    execute "my command" do
      command "/bin/echo 'Hello!'"
      environment ({'LC_ALL' => ''})
    end

I haven’t tried it, but you could also look at explicitly setting the LC_ALL environment variable to an empty value in your system level shell profile (/etc/profile), default environment setup files (e.g., /etc/environment), and/or digging into Ubuntu locale configuration.

From the Top!

At work, we’re currently using Chef to manage our systems and perform some application installations. A few weeks ago, we ran into a particularly strange head-scratcher that took me about half a day to track down.

Our application stack currently consists of Ubuntu 12.04 (precise), Apache 2.22, and PHP 5.3 running our integration of Drupal 7, vBulletin 4, and our own application code based on Zend Framework. We have a set of shell and PHP scripts that perform the installation of Drupal, VB, and our code all in one go. For all non-developer environments, Bamboo triggers chef-solo runs on code commits or when an engineer wants to redeploy to the manual testing servers.

The problem that popped up was that VB was having issues on the automated environments but only when run via chef-solo. When the script were run by hand on the same environment, everything worked like a charm.

At issue were some localization installation steps that were failing which caused the entire forum portion of the site to nearly completely fail to function. In the build logs, the only evidence was that SQL queries part of language installation were failing because a value wasn’t present but only for some languages.

I spent some time digging through our installation scripts and the VB code that was running the language instalation trying to understand where the issue came from. Of note, VB has a bunch of custom XML parsing code that uses iconv to attempt to handle character set conversions for localized phrases. I even got so far as to see the code improperly convert some of the French accented characters during the chef-solo runs that worked without issue when run manually.

Environment variable state was one of my first guesses when we has isolated the repro steps to chef-solo vs manual runs. The LC_* variables looked like a likely culprit but my initial inspection was cursory - I only paid attention to LANG and LC_CTYPE which were set correctly to en_US.utf8. Only after a lot more testing and research did I notice that LC_ALL was set to ‘C’ during the chef-solo run and was not set on the manual runs.

As soon as I explicitly set LC_ALL to the empty string in the chef cookbook, everything worked as expected. The iconv calls in the VB code now had the proper environment variables and weren’t trying to treat everything as the ‘C’ language type.

The underlying issue in Chef is the following code:

lib/chef/mixin/command/unix.rb, lines 53-58
1
2
3
4
5
6
    # Default on C locale so parsing commands output can be done
    # independently of the node's default locale.
    # "LC_ALL" could be set to nil, in which case we also must ignore it.
    unless args[:environment].has_key?("LC_ALL")
      args[:environment]["LC_ALL"] = "C"
    end

When LC_ALL is not in the environment variables at all, it defaults it to ‘C’. Ubuntu (and at least one version of debian that I had easy access to), LC_ALL is not defined at all:

Environment variables on Ubuntu 12.04 Precise
1
2
3
4
5
    $ env | grep -E 'LC|LANG'

    LANG=en_US.utf8
    LANGUAGE=en_US:
    LC_CTYPE=en_US.UTF-8

Interestingly, the locale command displays an empty LC_ALL:

Locale on Ubuntu 12.04 Precise
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
    $ locale

    LANG=en_US.utf8
    LANGUAGE=en_US:
    LC_CTYPE=en_US.UTF-8
    LC_NUMERIC="en_US.utf8"
    LC_TIME="en_US.utf8"
    LC_COLLATE="en_US.utf8"
    LC_MONETARY="en_US.utf8"
    LC_MESSAGES="en_US.utf8"
    LC_PAPER="en_US.utf8"
    LC_NAME="en_US.utf8"
    LC_ADDRESS="en_US.utf8"
    LC_TELEPHONE="en_US.utf8"
    LC_MEASUREMENT="en_US.utf8"
    LC_IDENTIFICATION="en_US.utf8"
    LC_ALL=

From the UNIX specification, we can see the effect of LC_ALL:

This variable determines the values for all locale categories. The value of the LC_ALL environment variable has precedence over any of the other environment variables starting with LC_ (LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, LC_TIME) and the LANG environment variable.

So, from now on I’ll likely explicitly set LC_ALL in Chef cookbooks as well as making sure that the environment variables exists but is empty as the default state of things in the OS.

References and Further Reading

There’s at least one ticket about this issue right now:

However, it looks like the ‘C’ default may be in place due to issues when they weren’t setting it at all:

Properly setting the locale for Ubuntu may also help alleviate problems. I haven’t tested this so I’m sure if it would set the proper environment variables or not

Comments