« September 2009 | Main | November 2009 »

October 2009

October 08, 2009

Cppref: reading cppreference.com docs offline, like man or info or perldoc

Today I created a tiny script called cppref, a wrapper for documents on cppreference.com.  Blurbs are:

  • docs are bundled with the interface, no network access required
  • works like man(1) or info(2) or perldoc(2)

It looks like follows.

$ cppref
You are here: C++ Reference

C++ Reference

                                  C++

General Topics                      * C++ Strings
                                    * C++ I/O
  * FAQ                                 + C++ String Streams
  * Pre-processor commands          * C++ Exceptions
  * Operator Precedence
  * Escape Sequences              C++ Standard Template Library (STL)
  * ASCII Chart
  * Data Types                      * Overview
  * Keywords                        * Iterators
                                    * C++ Algorithms
Standard C Library                  * C++ Vectors
                                    * C++ Double-Ended Queues
  * Overview                        * C++ Lists
  * Standard C I/O                  * C++ Priority Queues
  * Standard C String & Character   * C++ Queues
  * Standard C Math                 * C++ Stacks
  * Standard C Time & Date          * C++ Sets
  * Standard C Memory               * C++ Multisets
  * Other standard C functions      * C++ Maps
                                    * C++ Multimaps
                                    * C++ Bitsets

$ cppref vector
You are here: C++ Reference >> C++ Standard Template Library >> C++ Vectors

C++ Vectors

Vectors contain contiguous elements stored as an array.

Accessing members of a vector can be done in constant time, appending elements
to a vector can be done in amortized constant time, whereas locating a specific
value or inserting elements into the vector takes linear time.

Constructors create vectors and initialize them with some data
Operators    compare, assign, and access elements of a vector
assign       assign elements to a vector
at           returns an element at a specific location
back         returns a reference to last element of a vector
(snip)

Or if the specified term maps to multiple files,

$ cppref push_back        
multiple choices:
  stl::deque::push_back
  stl::list::push_back
  stl::vector::push_back
  string::push_back

By default, cppref uses w3m as its viewer, so you can follow the links to read the documents.

Cppref is available from search.cpan.org/dist/cppref or github.com/kazuho/cppref.  Hove fun!

October 07, 2009

Uploading an autotools-based distribution onto CPAN

Background

It is a pain to create binary packages.  But installing a program from source tarball is a tedious task.  You need to run ./configure & make && make install.  Sometimes you need to resolve the dependencies by hand as well.  That's where source-code-based package distribution systems come in, and the largest system is, IMHO, CPAN.  If you could upload a autotools-based distribution onto CPAN, then the users of the software can install them with the cpan command (or cpanp or cpanf or whatever), with the dependencies automatically resolved.

And for my case, it was considered especially benefitial, since the program I am now working on (it's called incline, a replicator for RDB shards using MySQL or PostgreSQL), uses perl scripts for running tests.  By distributing incline through CPAN, the perl modules required for running the test suite could be installed automatically.  Besides that, the expected users of the software outside my company are mostly perl users.

So I adjusted the files of incline so that it can be uploaded to CPAN (it continues to work as a normal autotools-based distribution as well).  The steps were as follows (for a working example, please see incline - search.cpan.org).

Step 1. write META.yml generator

CPAN packages should contain a file call META.yml which describes the metainformation of the package like dependencies, etc.  Since it would be a good idea to fill-in some fields of the file by reading other files of the distribution, I wrote a script file (META.yml.pl) that generates META.yml.  As you can see, it is pretty straightforward.  One point to note is that it has a --fix-makefile option that inserts dependency information into Makefile, which is mandatory for supporting old versions of the CPAN installer.

META.yml.pl
#! /usr/bin/perl

use strict;
use warnings;

use YAML;

my $meta = {
    name               => 'incline',
    abstract           => 'a replicator for RDB shards',
    version            => do {
        my $s = `echo VERSION | cpp -include src/incline_config.h`;
        $s =~ s/^.*\n\"([0-9_\.]+)\".*?$/$1/s
            or die "failed to obtain version number";
        $1;
    },
    author             => do {
        open my $fh, '<', 'AUTHORS'
            or die "failed to open AUTHORS:$!";
        my @authors = map {
            chomp $_;
            $_;
        } <$fh>;
        close $fh;
        \@authors;
    },
    license            => 'bsd',
    distribution_type  => 'script',
    dynamic_config     => 0,
    configure_requires => {
        perl          => 5.008,
        YAML          => 0,
    },
    requires           => {},
    build_requires     => {
        DBI                => 0,
        'List::MoreUtils'  => 0,
        'Scope::Guard'     => 0,
        'Test::mysqld'     => 0,
        'Test::postgresql' => 0,
    },
    resources          => {
        license => 'http://www.opensource.org/licenses/bsd-license.php',
    },
    no_index           => {
        directory => [ qw/example src t/ ],
        file      => [ qw/README.html/ ],
    },
    'meta-spec'        => {
        version => 1.4,
        url => 'http://module-build.sourceforge.net/META-spec-v1.4.html',
    },
    generated_by       => 'META.yml.pl',
};

if (@ARGV && $ARGV[0] eq '--fix-makefile') {

    my $prereq_expr = do {
        my %req = (
            %{$meta->{requires}},
            %{$meta->{build_requires}},
        );
        join ", ", map { "$_=>q[$req{$_}]" } sort keys %req;
    };
    print <<"EOT";
# MakeMaker Parameters:

#  PREREQ_PM => { $prereq_expr }

# --- MakeMaker post_initialize section:

EOT
    while (my $l = <STDIN>) {
        print $l;
    }

} else {

    print Dump($meta);

}

Step 2. call META.yml.pl from configure

As explained, older versions of the CPAN installer requires the dependencies to be written in Makefile (this is still the case for newer versions if your configuration script dynamically determines the dependencies).  In order to meet the requirement, I added a call to META.yml.pl at the end of my configure.ac.

configure.ac
AC_OUTPUT(Makefile src/Makefile)

cp Makefile Makefile.orig && perl META.yml.pl --fix-makefile < Makefile.orig > Makefile
if test "$?" -ne "0"
then
AC_MSG_ERROR([failed to add metainfo to Makefile])
fi

Step 3. adjust Makefile

There are a couple of things to adjust in the Maefile.  First, make test should be supported.  In my case, I already had one, that executes test scripts written in perl using Test::Harness.  The next thing is to include more than one POD (plain old document).  I already had one as well, since I generate man pages and readmes from POD.  The last thing is to generate and include two files in the distribution, META.yml and MANIFEST.  The Makefile uses a dependency and dist-hook to generate the files.  The files adde should be included in the EXTRA_DIST section of Makefile.am, of course.

Makefile.am
META.yml: META.yml.pl
perl $< > $@

dist-hook:
(cd $(distdir) && find * -type f) > $(distdir)/MANIFEST

test:
perl -MTest::Harness -we 'runtests(@ARGV)' t/*.t

Step 4. write Makefile.PL

Create Makefile.PL that calls ./configure.  Do not write any configuration logic in Makefile.PL, all logic should go into configure.ac so that the distribution could be built by using either Makefile.PL or configure.

Makefile.PL
exec './configure', @ARGV;
die "failed to execute configure:$!";

Conclusion

As described, it is not (so) difficult to add CPAN compatibility to existing autotools-based distributions, or to create a new distribution that works as both.  But to tell the truth it was not as easy as it seems to figure out the right way.  It would have been impossible without help from charbar, migayawa, tokuhirom, and walf443.  I would like to thank them all.

October 05, 2009

Apache で X-Reproxy-URL ヘッダを使えるようにするモジュール mod_reproxy を書いた

ウェブアプリケーションにおいて、認証がかかっている画像や大きなファイルを配信する場合には、Perlbal 等でサポートされている X-Reproxy-URL ヘッダが有効なことが知られていて、その理由としては、

  • (メモリを大食いする) アプリケーションサーバのプロセスを転送終了まで占有しない
  • HTTP ベースの分散ファイルシステムとリバースプロキシが直接交信するので、ネットワーク負荷が低い

といった点が挙げられます。「でも、Apache は X-Reproxy-URL ヘッダをサポートしてないんだよねー」という話が、先日の YAPC::Asia 2009 においても話題になっていました[要出典]。回避策としては、ワンタイムURLのような手法もあるのですが、セキュリティな懸念もあります。

なんとかしたいなと思っていたのですが、気が向いたので、Apache に X-Reproxy-URL ヘッダ対応機能を追加するモジュール mod_reproxy を書いてみました。Github に置いてあります。

kazuho's mod_reproxy at master - GitHub

使い方は簡単。ダウンロードして、apxs を使って、コンパイル&インストールを行います。

# apxs -i -a -c -Wc,-Wall -Wc,-g -Wc,-O2 -lcurl mod_reproxy.c

その後、Apache の設定ファイルに

Reproxy On

と追記すれば、その追記した範囲において、X-Reproxy-URL ヘッダによる内部リダイレクトが有効になります。

それだけです。特に説明することもないのですが、状況によっては便利に使えるかと思います。

参考: lighttpd の X-Sendfile を使えるApacheモジュール - spiritlooseのはてなダイアリー