ngen

Build file generator (engine) for the Ninja build system.

Licensed under the GPLv3.

Methodology and Overview

The problem with existing meta build systems---or makefile generators---like Meson and CMake is that they are needlessly complex for small-to-medium-sized C/C++ projects. These projects are just large enough that hand-writing a makefile would be painful, but trying to configure a large meta build system would be just as painful. It is unacceptable that building a simple project with 10 .c files and a library requires knowledge of a unique, obscure configuration language which is constantly changing.

ngen aims to make generating build files for these small C/C++ projects as simple as possible by using a basic key-value configuration file using patterns that any experienced programmer should be familiar with. In doing so, ngen fills the gap between writing your own makefile and wrangling with CMakeLists.txt.

ngen generates files for the small and modern Ninja build system. "Where other build systems are high-level languages Ninja aims to be an assembler," according to Ninja's website. It can be thought of as a simpler, faster replacement for the classic make. It is used by default by Meson; CMake can also be configured to use ninja as a backend. Ninja is used by used by Google to build Chromium, v8, etc., and it is also used to build LLVM. (All this to say, Ninja is commonly used and it is likely installed on your system already.)

Building

Build: cargo build --release

Install: sudo sh install.sh

Uninstall: sudo sh uninstall.sh

Usage

What follows is a tutorial of how to set up ngen for an existing executable project.

Basics

The first thing you will need is an ngen.toml file. This is what will specify all of the build parameters, such as compilation flags, source files to track, etc. Start by creating this file and opening it in your editor.

The simplest valid ngen.toml just lists the source files you want to track. Lets say you have a executable project that looks like this:

example
├── ngen.toml
└── src
    ├── foobar.c
    ├── functions.c
    ├── include
    │   ├── foobar.h
    │   ├── functions.h
    │   ├── main.h
    │   └── util.h
    ├── main.c
    └── util.c

In your ngen.toml, write the following:

[targets.main]
sources = [
  "src/main.c",
  "src/util.c",
  "src/functions.c",
  "src/foobar.c",
]

The sources key is a list of strings, each specifying a single source file name.

Now run ngen. This will generate a build.ninja file in the current working directory. You won't ever have to touch this file; that's what ngen is for. You also won't ever have to run ngen yourself again (unless your build.ninja gets deleted); Ninja will take care of regenerating the build file if ngen.toml changes.

With your build.ninja generated, run ninja on the command line. That's it! Your project is now built, you will find the executable at build/main/a.out. Remember, you can also freely add and remove files from the above list without running ngen again: Ninja will regenerate the build.ninja for you.

Now, while this is functional, it isn't very useful. It is very likely that you will want to specify a compiler (gcc/clang), pass some flags, link some libraries into your final executable, and definitely name your program something other than "a.out." ngen makes these things dead simple, too.

The outfile key is a string that specifies the name of the file produced by the linker (see below).

Lets set this to "example."

outfile = "example"

The compiler key is a string that specifies the program that will be used to turn .c files into .o files.

By default, if not specified, compiler is set to "cc," which on Linux systems should be a C compiler. It should be noted that the compiler you choose must support the -MD and -MF flags to generate dependency files (both gcc and clang support this). Lets say we want to use "gcc." Add the following line to your ngen.toml:

compiler = "gcc"

The compiler_flags key is a list of strings that contains the arguments to be passed to the compiler during the compilation of each source file.

It is not necessary to add the -c or -o outfile flags, ngen will take care of this for you. For example, add the following to your ngen.toml:

compiler_flags = ["-Wall", "-Wextra -O2"]

The linker key is a string that specifies the program that will be used to combine the .o files into the final outfile.

If the linker key is not found, it will be set to the value of compiler. For this example, we don't have to change anything here.

The linker_flags key is a list of strings that contains the arguments to be passed to the linker during the linking of the outfile.

Library flags (-lm, -lyourlib) should NOT be included here. This is for linker options, not libraries. The syntax is the same as compiler_flags. There is nothing we have to set here.

The linker_libs key is a list of strings that contains the link library arguments to be linked to the outfile.

THIS is where library flags (-lm, -lyourlib) go. Lets say our example project needs the math library:

linker_libs = ["-lm"]

Now, our ngen.toml looks like this:

[targets.main]
outfile = "example"
compiler = "gcc"
compiler_flags = ["-Wall", "-Wextra -O2"]
linker_libs = ["-lm"]
sources = [
  "src/main.c",
  "src/util.c",
  "src/functions.c",
  "src/foobar.c",
]

This is a much more realistic looking project. Once again, any changes to any of these values will be automatically picked up by Ninja and accounted for in the build. Running ninja -v immediately after saving ngen.toml should show that the options you set were recognized, and your files were rebuilt accordingly.

Now, this is a good start. But, it often the case that, in a project, you want to have multiple different build targets, or configurations, that build the project with slightly different parameters; for example, it is common to have a "debug" build target that is unoptimized and includes debugging symbols, and a "release" build that is optimized at compile time. ngen is designed to make this configuration as easy as possible.

Targets

When we were specifying parameters above, we were doing so in the main target [targets.main]. A "target" is a self-contained build process that builds its outfile from the sources and other paramenters provided. We could have named this target whatever we wanted, but the target named main is special, as we shall presently see. Lets create a new target called debug. Add the following to your ngen.toml:

[targets.debug]
outfile = "example_dbg"
compiler_flags = ["-g"]

What's going on here? How does debug know what files to operate on, what compiler to use, etc? Well, the main target is special: all targets inherit the parameters set in the main target. Inheritance works according to two simple rules: arrays append, strings supercede.

The first thing that happens is debug takes on all the same parameters from main. Then, ngen reads the outfile key in debug. Becuase outfile is a string, debug.outfile is overwritten as the value specified, in this case "example_dbg". On the other hand, since compiler_flags is a list, the elements sepecified in debug.compiler_flags are appended to the list of flags specified in main. So in this case, the effective value of debug.compiler_flags is ["-Wall", "-Wextra -O2", "-g"].

For our debug build, we probably don't want the -O2 flag---optimizations should only happen in a "release" type build. So, lets remove the -O2 flag from main and place it in a new target called release. Our ngen.toml should now look like this:

[targets.main]
outfile = "example"
compiler = "gcc"
compiler_flags = ["-Wall", "-Wextra"]
linker_libs = ["-lm"]
sources = [
  "src/main.c",
  "src/util.c",
  "src/functions.c",
  "src/foobar.c",
]

[targets.debug]
outfile = "example_dbg"
compiler_flags = ["-g"]

[targets.release]
compiler_flags = ["-O2"]

This brings up an important design pattern you should keep in mind when writing your ngen.toml: the special main target should only contain the largest subset of all your build parameters. Additonal targets should add specific parameters for specific use cases, as we saw in the above example.

In a nutshell, inheritance allows you to easily create multiple targets with small variations, without having to rewrite the same thing over and over again. You can always disable inheritance using the opts.inherit = false key on targets that you do not want to inherit from main. You can also change the parent target that a target inherits from using the opts.inherit_from = "target" key, replacing target with the name of the desired parent target.

Save ngen.toml, and try running ninja -v debug or ninja -v release. You should see that each of these targets uses the parameters that we specified with inheritance. Outfiles for a given target are always placed in build/<target_name>/, as you can see with tree build. This keeps things organized, and also means we don't have to specify a different outfile name for each target (I did above just to show you how strings are replaced in the inheritance system).

Note that by default, running ninja alone with no target specifed will run every single target it finds. You can change this behavor by adding the opts.default = true key to the targets you want to be built when Ninja is invoked with no arguments. Say that this example project is under active developemnt, and you will be building the debug target alot. You can add the opts.default = true flag to the [debug] table, and now running ninja by itself will only build the debug target. You can still build the release and main targets by running ninja release and ninja main.

It is easy to see how powerful this simple configuration file is already. However, ngen has a few more features that you may find useful.

Configuration

The config table is where you can specify certain options which change the way ngen behaves. One useful feature is generating a compile_commands.json file for the clangd LSP.

To enable the generation of compile_commands.json, simply add the following line to your ngen.toml:

[config]
compile_commands = true

The next time you run Ninja, ngen will automatically generate build/compile_commands.json where it can be picked up by clangd. Ninja will make sure that this file is kept up to date as well. Just set the option in your ngen.toml and forget about it.

By default, the compile_commands.json will be generated according to the build specs of the main target. To change which target it is generated for, use the config.compile_commands_target key. For example,

[config]
compile_commands = true
compile_commands_target = "debug"

will generate the the compile_commands for the debug target.

Our final ngen.toml looks like this:

[config]
compile_commands = true
compile_commands_target = "debug"

[targets.main]
outfile = "example"
compiler = "gcc"
compiler_flags = ["-Wall", "-Wextra"]
linker_libs = ["-lm"]
sources = [
  "src/main.c",
  "src/util.c",
  "src/functions.c",
  "src/foobar.c",
]

[targets.debug]
opts.default = true
outfile = "example_dbg"
compiler_flags = ["-g"]

[targets.release]
compiler_flags = ["-O2"]

Advanced Usage

Lets say that your project also has a library in the lib folder, with two files, lib.c and functions.c. You want to compile these into a libexample.a file so that you can link them to your main executable. Add a new target called lib:

[targets.lib]
opts.inherit = false
outfile = "libexample.a"
compiler = "gcc"
linker = "ar"
linker_flags = "r"
sources = [
  "lib/lib.c",
  "lib/functions.c",
]

If you try to run this as is, you will see that there is a problem with the link step. ngen constructs commands according to a certain format to make configuration easier. The format for the link step is {linker} {linker_flags} -o {outfile} {objects} {linker_libs}. This means that currently, the above configuration is trying to run ar r -o libexample.a <object files>. ar does not take an -o flag. We can change this format string to properly produce the library. Add the following entry to the lib target above:

opts.link_cmd_fmt = "{linker} {linker_flags} {outfile} {objects} {linker_libs}"

By removing the -o flag, the library should "link" correctly now. But, we could simplify this configuration slightly by removing some of those unnecessary format placeholders. Here is an alternate form that should work exactly the same:

[targets.lib]
opts.inherit = false
opts.link_cmd_fmt = "ar r {outfile} {objects}"
outfile = "libexample.a"
compiler = "gcc"
sources = [
  "lib/lib.c",
  "lib/functions.c",
]

An option opts.compile_cmd_fmt also exists. Its default is {compiler} {compiler_flags} -MD -MF {depfile} -o {object} -c {source}. Note that if you remove the -MD -MF {depfile} component, ninja will have no way to discover what header files your source file depends on.