Tanguy Ortolo - Debian

Bad support of ZIP archives with extra fields

Tanguy — Tue, 21 Mar 2017 19:33:00 +0000

For sharing multiple files, it is often convenient to pack them into an archive, and the most widely supported format to do so is probably ZIP. Under *nix, you can archive a directory with Info-ZIP:

% zip -r something.zip something/

(When you have several files, it is recommended to archive them in a directory, to avoid cluttering the directory where people will extract them.)

Unsupported ZIP archive

Unfortunately, while we would expect ZIP files to be widely supported, I found out that this is not always the case, and I had many recipients failing to open them under operating systems such as iOS.

Avoid extra fields

That issue seems to be linked to the usage of extra file attributes, that are enabled by default, in order to store Unix file metadata. The field designed to store such extra attributes was designed from the beginning so each implementation can take into account attributes it supports and ignore any other ones, but some buggy ZIP implementation appear not to function at all with them.

Therefore, unless you actually need to preserve Unix file metadata, you should avoid using extra fields. With Info-ZIP, you would have to add the option -X:

% zip -rX something.zip something/

Generate man pages for awscli

Tanguy — Wed, 23 Nov 2016 17:25:00 +0000

No man pages, but almost

The AWS Command Line Interface, which is available in Debian, provides no man page. Instead, that tool has an integrated help system, which allows you to run commands such as aws rds help, that, for what I have seen, generates some reStructuredText, then converts it to a man page in troff format, then calls troff to convert it to text with basic formatting, and eventually passes it to a pager. Since this is close to what man does, the result looks like a degraded man page, with some features missing such as the adaptation to the terminal width.

Well, this is better than nothing, and better than what many under-documented tools can offer, but for several reasons, it still sucks: most importantly, it does not respect administrators' habits and it does not integrate with the system man database. You it does not allow you to use commands such as apropos, and you will get no man page name auto-completion from your shell since there is no man page.

Generate the man pages

Now, since the integrated help system does generate a man page internally, we can hack it to output it, and save it to a file:

Description: Enable a mode to generate troff man pages
 The awscli help system internally uses man pages, but only to convert
 them to text and show them with the pager. This patch enables a mode
 that prints the troff code so the user can save the man page.
 .
 To use that mode, run the help commands with an environment variable
 OUTPUT set to 'troff', for instance:
     OUTPUT='troff' aws rds help
Forwarded: no
Author: Tanguy Ortolo <tanguy+debian@ortolo.eu>
Last-Update: 2016-11-22

Index: /usr/lib/python3/dist-packages/awscli/help.py
===================================================================
--- /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
+++ /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
@@ -49,6 +49,8 @@
     Return the appropriate HelpRenderer implementation for the
     current platform.
     """
+    if 'OUTPUT' in os.environ and os.environ['OUTPUT'] == 'troff':
+        return TroffHelpRenderer()
     if platform.system() == 'Windows':
         return WindowsHelpRenderer()
     else:
@@ -97,6 +99,15 @@
         return contents


+class TroffHelpRenderer(object):
+    """
+    Render help content as troff code.
+    """
+
+    def render(self, contents):
+        sys.stdout.buffer.write(publish_string(contents, writer=manpage.Writer()))
+
+
 class PosixHelpRenderer(PagingHelpRenderer):
     """
     Render help content on a Posix-like system.  This includes

This patch must be applied from the root directory with patch -p0, otherwise GNU patch will not accept to work on files with absolute names.

With that patch, you can run help commands with an environment variable OUTPUT='troff' to get the man page to use it as you like, for instance:

% OUTPUT='troff' aws rds help > aws_rds.1
% man -lt aws_rds.1 | lp

Generate all the man pages

Now that we are able to generate the man page of any aws command, all we need to generate all of them is a list of all the available commands. This is not that easy, because the commands are somehow derived from functions provided by a Python library named botocore, which are derived from a bunch of configuration files, and some of them are added, removed or renamed. Anyway, I have been able to write a Python script that does that, but it includes a static list of these modifications:

#! /usr/bin/python3

import subprocess
import awscli.clidriver


def write_manpage(command):
    manpage = open('%s.1' % '_'.join(command), 'w')
    command.append('help')
    process = subprocess.Popen(command,
            env={'OUTPUT': 'troff'},
            stdout=manpage)
    process.wait()
    manpage.close()


driver = awscli.clidriver.CLIDriver()
command_table = driver._get_command_table()

renamed_commands = \
    {
        'config': 'configservice',
        'codedeploy': 'deploy',
        's3': 's3api'
    }
added_commands = \
    {
        's3': ['cp', 'ls', 'mb', 'mv', 'presign', 'rb', 'rm', 'sync',
               'website']
    }
removed_subcommands = \
    {
        'ses': ['delete-verified-email-address',
                'list-verified-email-addresses',
                'verify-email-address'],
        'ec2': ['import-instance', 'import-volume'],
        'emr': ['run-job-flow', 'describe-job-flows',
                'add-job-flow-steps', 'terminate-job-flows',
                'list-bootstrap-actions', 'list-instance-groups',
                'set-termination-protection',
                'set-visible-to-all-users'],
        'rds': ['modify-option-group']
    }
added_subcommands = \
    {
        'rds': ['add-option-to-option-group',
                'remove-option-from-option-group']
    }

# Build a dictionary of real commands, including renames, additions and
# removals.
real_commands = {}
for command in command_table:
    subcommands = []
    subcommand_table = command_table[command]._get_command_table()
    for subcommand in subcommand_table:
        # Skip removed subcommands
        if command in removed_subcommands \
                and subcommand in removed_subcommands[command]:
            continue
        subcommands.append(subcommand)
    # Add added subcommands
    if command in added_subcommands:
        for subcommand in added_subcommands[command]:
            subcommands.append(subcommand)
    # Directly add non-renamed commands
    if command not in renamed_commands:
        real_commands[command] = subcommands
    # Add renamed commands
    else:
        real_commands[renamed_commands[command]] = subcommands
# Add added commands
for command in added_commands:
    real_commands[command] = added_commands[command]

# For each real command and subcommand, generate a manpage
write_manpage(['aws'])
for command in real_commands:
    write_manpage(['aws', command])
    for subcommand in real_commands[command]:
        write_manpage(['aws', command, subcommand])
                         'sync', 'website']}

This script will generate more than 2,000 man page files in the current directory; you will then be able to move them to /usr/local/share/man/man1.

Since this is a lot of man pages, it may be appropriate to concatenate them by major command, for instance all the aws rds together…

Process command line arguments in shell

Tanguy — Wed, 08 Jun 2016 13:29:00 +0000

When writing a wrapper script, one often has to process the command line arguments to transform them according to his needs, to change some arguments, to remove or insert some, or perhaps to reorder them.

Naive approach

The naive approach to do that is¹:

# Process arguments, building a new argument list
new_args=""
for arg in "$@"
do
    case "$arg"
    in
        --foobar)
            # Convert --foobar to the new syntax --foo=bar
            new_args="$args --foo=bar"
        ;;
        *)
            # Take other options as they are
            new_args="$args $arg"
        ;;
    esac
done

# Call the actual program
exec program $new_args

This naive approach is simple, but fragile, as it will break on arguments that contain a space. For instance, calling wrapper --foobar "some file" (where some file is a single argument) will result in the call program --foo=bar some file (where some and file are two distinct arguments).

Correct approach

To handle spaces in arguments, we need either:

to quote them in the new argument list, but that requires escaping possible quotes they contain, which would be error-prone, and implies using external programs such as sed;
to use an actual list or array, which is a feature of advanced shells such as Bash or Zsh, not standard shell…

… except standard shell does support arrays, or rather, it does support one specific array: the positional parameter list "$@"². This leads to one solution to process arguments in a reliable way, which consists in rebuilding the positional parameter list with the built-in command set --:

# Process arguments, building a new argument list in "$@"
# "$@" will need to be cleared, not right now but on first iteration only
first_iter=1
for arg in "$@"
do
    if [ "$first_iter" -eq 1 ]
    then
        # Clear the argument list
        set --
        first_iter=0
    fi
    case "$arg"
    in
        --foobar) set -- "$@" --foo=bar ;;
        *) set -- "$@" "$arg" ;;
    esac
done

# Call the actual program
exec program "$@"

Notes

I you prefer, for arg in "$@" can be simplified to just for arg.↑
As a reminder, and contrary to what it looks like, quoted "$@" does not expand to a single field, but to one field per positional parameter. ↑

Let's Encrypt: threat or opportunity to other certificate authorities?

Tanguy — Fri, 15 Apr 2016 13:25:00 +0000

Let's Encrypt is a certificate authority (CA) that just left beta stage, that provides domain name-validated (DV) X.509 certificates for free and in an automated way: users just have to run a piece of software on their server to get and install a certificate, resulting in a valid TLS setup.

Let's Encrypt logo

A threat to other certificate authorities

By providing certificates for free and automatically, Let's Encrypt is probably a threat a other CAs, a least for part of their activity. Indeed, for people that are satisfied with DV certificates, there are not many reasons to pay a commercial CA to get certificates in a non-automated way. For the CAcert non-commercial CA, that may mean a slow death, as this is their main activity¹.

For people that want organization-validated (OV) or extended validation (EV) certificates, Let's Encrypt is not suitable, so it will not change anything regarding that.

An opportunity for the most reactive

The entrance of Let's Encrypt is also a significant opportunity for the certificate authorities that will be reactive enough to take advantage of their innovation. Indeed, they introduced automation in both domain name validation and certificate issuance (and revocation), by defining an open protocol that is meant to become an Internet standard. That protocol, named ACME, is not tied to Let's Encrypt and has several free software implementations, so it could be used for the same purpose by commercial CAs.

A certification authority could, for instance:

ask the customer to provision some pre-paid account;
manually validate the customer's identity once;
allow the customer to register using ACME, and associate that registration to his validated identity;
allow the customer to get organization-validated, or perhaps even extended validation certificates using ACME, and making corresponding debits to his pre-paid account.

Such processes may require or benefit from improvements of the ACME protocol, which is the very reason Internet standards are defined in an open way.

The first certification authority that would implement such a process could gain an advantage over its competitors, as it would greatly simplify getting and renewing certificates. I think even Let's Encrypt people would be happy to see that happen, as it would serve their goal, that is basically to help securing the Internet! Personally, I could buy such a service (assuming it is not restricted to juridical persons, according to a quite common (and detestable) sale discrimination against natural persons²).

Notes

CAcert is an unrecognised certificate authority, that provides an identity validation through a web of trust, and issues DV server certificates that do not include the validated identity. Now that Let's Encrypt can issue valid DV certificates, CAcert is no longer relevant for that activity. It also issues personal certificates, that do include the validated identity, and that can be used for encryption (e.g. S/MIME), signing (e.g. code signing) or authentication, which is an activity Let's Encrypt does not compete with.
Yes, the Organization field of a certificate is probably not relevant to indicate a physical person's name, but the CommonName field is. Yes, that field is usually abused to store the domain name, but a proper use would be to put the owner's name in the CommonName field, and the domain names in the subjectAltName field.↑

Removing sam2p from Debian

Tanguy — Fri, 22 Jan 2016 00:52:00 +0000

Issues with sam2p and removal

I have been maintaining the Debian package of sam2p for some time. Unfortunately, the upstream development of that program is no longer active, and it is using an old custom build chain that no longer works with recent version of GCC.

This package is currently failing to build from source, and while I have been able to patch some issues in the past, and it may still be possible to fix it again, this is not really sustainable.

I am therefore considering to remove sam2p from Debian, unless someone has a very good reason to keep it and is able and willing to maintain it.

Alternative

sam2p is a raster image conversion tool that can convert PNG and JPEG to EPS and PDF while keeping their compression, which is mostly useful to use them in documents compiled with LaTeX. Fortunately, the same can be done with ImageMagick. If you want to convert to EPS, you have to specify that you want EPS 2 or 3, otherwise it would produce EPS level 1 which does not provide native raster compression:

% convert debian-openlogo-raster100.png \
          eps3:debian-openlogo-raster100.eps
% convert debian-openlogo-raster100.png \
          debian-openlogo-raster100.pdf
% ls -lh
1.7K debian-openlogo-raster100.png
6.0K debian-openlogo-raster100.eps
8.8K debian-openlogo-raster100.pdf

% convert photograph.jpg eps3:photograph.eps
% convert photograph.jpg photograph.pdf
% ls -lh
657K photograph.jpg
662K photograph.eps
664K photograph.pdf

% convert scanned-document.png eps3:scanned-document.eps
% convert scanned-document.png scanned-document.pdf
140K scanned-document.png
145K scanned-document.eps
150K scanned-document.pdf

This is a bit less efficient than sam2p, as convert seems to add some fixed overhead, but it does keep the appropriate compression algorithm. See this documentation page from ImageMagick for more information.

Using appropriate formats

As a reminder, when writing LaTeX documents, depending on your build chain, you can use:

photographs: JPEG or EPS (converted from JPEG with ImageMagick);
raster drawings, screenshots…: PNG or EPS (converted from PNG with ImageMagick);
vector graphics: PDF or EPS (convertes from SVG with Inkscape).

Scale manufacturers…

Tanguy — Mon, 26 Jan 2015 14:54:00 +0000

Dear manufacturers of kitchen scales, could you please stop considering your clients as idiots, and start developing useful features?

Liquid measurement: this is one feature that is available on almost every electronic scale available. Except it is completely useless to people that use the metric system, as all it does is replace the usual display in grammes by centilitres and divide the number on display by ten. Thank you, but no person that has been to school in a country that uses the metric system needs electronic assistance to determine the volume corresponding to a given weight of water, and for people that have not, a simple note written on the scale, stating that “for water or milk, divide the weight in grammes by ten to get the volume in centilitres” should be enough.

Now, there is still one thing that an electronic scale could be useful for, which is determining the volume of liquids other than water (density 1 g/ml) or milk (density approx. equal to 1 g/ml), most importantly: oil (density approx. equal to .92 g/ml for edible oils like sunflower, peanut, olive and canola).

Proof of address: use common sense!

Tanguy — Thu, 08 Jan 2015 12:54:00 +0000

As I have just moved to a new home, I had to declare my new address to all my providers, including banks and administrations which require a proof of address, which can be a phone, DSL or electricity bill.

Well, this is just stupid, as, by definition, one will only have a bill after at least a month. Until then, that means the bank will keep a false address, and that the mail they send may not be delivered to the customer.

Now, bankers and employees of similar administrations, if you could use some common sense, I have some information for you: when someone moves to a new home, unless he is hosted by someone else, he is either renter or owner. Well, you should now that a renter has one contract that proves it, which is called a lease. And an owner has one paper that proves it, which is called a title, or, before it has been issued by administration, a certificate of sale. Now if you do not accept that as a proof of address, you just suck.

Besides, such a zeal to check one's address is just pointless, as it is just to get a proof of address without waiting for a phone, DSL or electricity bill (or to prove a false address, actually…) by just faking one. And as a reminder, at least in France, forgery is punishable by law but defined as an alteration of truth which can cause a prejudice, which means modifying a previous electricity bill to prove your actual address is not considered as a forgery (but using the same mean to prove a false address is, of course!).

Using bsdtar to change an archive format

Tanguy — Tue, 09 Dec 2014 16:00:00 +0000

Streamable archive formats

Archive formats such as tar(5) and cpio(5) have the advantage of being streamable, so you can use them for transferring data with pipes and remote shells, without having to store the archive in the middle of the process, for instance:

$ cd public_html/blog
$ rgrep -lF "archive" data/articles \
      | pax -w \
      | ssh newserver "mkdir public_html/blog ;
                       cd public_html/blog ;
                       pax -r"

Turning a ZIP archive into tarball

Unfortunately, many people will send you data in non-streamable archive formats such as ZIP¹. For such cases, bsdtar(1) can be useful, as it is able to convert an archive from one format to another:

$ bsdtar -cf - @archive.zip \
      | COMMAND

These arguments tell bsdtar to:

create an archive;
write it to stdout (contrary to GNU tar which defaults to stdout, bsdtar defaults to a tape device);
put into it the files it will find in the archive archive.zip.

The result is a tape archive, which is easier to manipulate in a stream than a ZIP archive.

Notes

Some will say that although ZIP is based on an file index, it can be stream because that index is placed at the end of the archive. In fact, that characteristic only allows to stream the archive creation, but requires to store the full archive before being able to extract it. ↑.