Differences between revisions 9 and 10
Revision 9 as of 2005-07-14 15:16:56
Size: 12995
Editor: JohnVk
Comment:
Revision 10 as of 2005-07-14 19:17:12
Size: 13322
Editor: JohnVk
Comment:
Deletions are marked like this. Additions are marked like this.
Line 19: Line 19:
 * Even tho std_env has an override system, for any user's program with 3 levels of hierarchy or more, it is prohibitive to have to check out every level just to build with a new version of a leaf.
Line 23: Line 24:
 * The version selection feature needs to be robust/clean/versatile enuf and easy enuf to use to release to the outside world.

Version Selection

Some SDL models which builds use in turn import other models. A good example is std_env which imports versions of different bridges, compilers and other tools, libraries, etc. Such models usually allow the caller to specify overrides as parameters to change the versions of these sub-components which get used. To see how this works, look at:

See also [http://www.vestasys.org/doc/man/html/voverrides.5.html#TransPkgOvsSect "Transient Package Overrides" in the voverrides(5) man page].

The Problem

The handling of version overrides has been written in SDL code. There are several problems with this:

  • Even tho std_env has an override system, for any user's program with 3 levels of hierarchy or more, it is prohibitive to have to check out every level just to build with a new version of a leaf.
  • Other, non-SDL programs want to understand the currently selected version of different components. For example, the user is often interested in asking "which version of component X is my build using?" Only the SDL code knows which version is actually being used, and only at run-time. You can write another program which parallels the SDL override logic, but that's ugly, prone to errors, and becomes a maintenance headache.
  • Other, non-SDL programs want to change the currently selected version of different components. For example, you might want to automatically generate overrides for new versions of certain components, possibly based on a criteria such as attributes. vupdate can't do that unless you're directly changing the model which already imports them, which you usually aren't with overrides. Another possibility is writing a GUI that allows the user to change the versions they have selected.

  • It has led to multiple subtly different forms of version overrides, which can be confusing for users
  • Dont return all sources back to the top level. Rather pool of un-evaluated model funcs.
  • The version selection feature needs to be robust/clean/versatile enuf and easy enuf to use to release to the outside world.

Version selection is a fairly simple problem. While it is possible to implement it in SDL, you don't need all the flexibility that SDL gives you to solve the problem of version selection.

New version selection

Vesta builds would be easier to configure if the selection of which versions to use (overrides, etc.) was separated from the SDL code that defines the build flow. Removing the selection of versions from the flexibility of a full programming language would make writing tools that understand/generate a configuration much easier.

Most of the key features of how improt-based version selection works should be preserved:

  • only specific versions (not references to "latest")
  • only taken from a versioned source file
  • versioned like other source files
  • take base config, add changes

The main thing that should be changed is the use of SDL code to pick the versions to be used. SDL code will obviously still need to get access to the selected versions, but the concept of any variable containing overrides should be eliminated. SDL code should never be checking to see if an override is present and choosing to use it instead of some other version. Instead, we should have simple, well-defined versuib selection semantics so non-SDL programs can understand & manipulate the versions used by a build.

Tricky parts:

  • may need to use different versions of the same thing in different parts of a builds (e.g. different versions of a shared library during different tool invocations)

SDL still needs some way to access the selected versions of different components. This could be done through a new import-like syntax. Another possibility would be feeding the version selection to the top-level model as its initial dot. Either way, we're creating a new namespace from which the SDL will somehow get the selected versions.

We need to associate the version selection file with the top-level model. (We wouldn't want the user to have to specify both at build time.) Maybe name the version selection file the same as the model, only ending in ".vsel".

Version Selection Files

What might a version selection file look like? It could be essentially the same as the current [http://www.vestasys.org/doc/sdl-ref/models.html#import imports clause] of a SDL model. An imports clause defines a hierarchical namespace of variables referring to SDL models from existing immutable versions, and we need the same capability.

There's two significant extensions we'd need:

  1. A method to include another version selection file and start with its namespace as a basis.

  2. A method to alter existing portions of the namespace. Imagine that the variables created by an included version selection file are a binding. Now use ++ to overlay the new imported names on top of that. In fact, maybe each assignment statement should effectively be ++ed into the namespace built up so far.

So, for example, the version selection file in a std_env package might look like this:

import
  // The std_env model is the build.ves here in the std_env package
  std_env = build.ves;

from /vesta/vestasys.org/bridges import
  // Versions of standard bridges to use
  bridges = [ generics/2,
              c = c_like/4,
              cxx = c_like/4,
              lex/1,
              yacc/1,
              lim/1,
              mtex/1 ];

from /vesta/vestasys.org/libs import
  // Versions of libraries to use
  libs/c = [
    gc/6.4/2,
    z = zlib/1.2.2/1,
  ];

Then an actual package using this might have a version selection file like this:

include /vesta/vestasys.org/platforms/linux/debian/i386/std_env.sarge/1/top.vsel;

// Use a newer version of the c_like bridge
from /vesta/vestasys.org/bridges import
  bridges = [ c = c_like/checkout/5/3,
              cxx = c_like/checkout/5/3 ];

// Use a newer version of zlib

from /vesta/vestasys.org/libs import
  libs/c/z = zlib/1.2.3/1;

This would overlay versions for three names in the version selection namespace:

  1. bridges/c

  2. bridges/cxx

  3. libs/c/z

Later, some SDL code would access these names and use their contents. All overriding happens in the version selection file, so SDL code doesn't need any concept of override parameters/variables.

Using Selected Versions

How do individual SDL models use the versions selected through this new machanism?

First of all, they would not use the existing [http://www.vestasys.org/doc/sdl-ref/models.html#import imports syntax]. That can only be used to refer to specific versions, but what we really need is a way to parameterize an SDL model in terms of the namespace of selected versions. (An SDL model might use normal imports for other models local to the directory where it exists, but it wouldn't use the from syntax of absolute paths for models.)

Probably the best way to do this would be to add some new syntax which allows a model to get imports out of the namespace defined by the version selection file selected when the build starts. This could be an entirely new keyword like this:

import
  // Local import
  self = build.ves;
use
  // Take a value from the version selection namespace
  std_env;
{
  . = std_env()/env_build();

  return self();
}

Or we could extend the import clause syntax to make the version selection namespace accessible as though it were a subdirectory with a special name, maybe some unused non-alphanumeric chaaracter:

import
  // Local import
  self = build.ves;
  // Take a value from the version selection namespace
  std_env = @/std_env;
{
  . = std_env()/env_build();

  return self();
}

Either way, the important thing is that the idnvidual SDL model doesn't specify any version for std_env here. It simply uses whatever version the user has selected.

Can we do this without new features?

Would it be possible to set up something that has these properties with the current implementation? Mostly. How would we do so?

  • We already have a hierarchical namespace which different models have access to: the special variable dot (.). We could store the hierarchy of selected versions in it. We would want to keep it along side the bridges and other pieces of build environment normally placed in dot, so we would want to put it in a sub-binding just for this purpose. Perhaps something like ./_versions_, although that's not really a great name as it holds models rather than versions, so maybe ./_models_ or ./_selection_, although I'm not crazy about either of those.

  • When a model wants the model for something else, it just pulls it out of ./_models_ (or whatever we call it).

  • The top-level model imports a single model which defines ./_models_. This model may import one or more others for base configurations and ++ their results together. It may also ++ in other bindings it creates from imports locally for version overrides.

  • When another program wants to know the selected versions of everything, it evaluates this version selection model. It's result will be a hierarchy (in the form of a binding) of selected versions (in the form of models). The printed result of the evaluator will show this hierarchy of seloected versions.

  • When another program wants to add a version override, it re-writes the version selection model used by the top-level model.

What would this look like for the examples above?

A version selection model for std_env:

import
  // The std_env model is the build.ves here in the std_env package
  std_env = build.ves;

from /vesta/vestasys.org/bridges import
  // Versions of standard bridges to use
  bridges = [ generics/2,
              c = c_like/4,
              cxx = c_like/4,
              lex/1,
              yacc/1,
              lim/1,
              mtex/1 ];

from /vesta/vestasys.org/libs import
  // Versions of libraries to use
  libs = [ c = [
    gc/6.4/2,
    z = zlib/1.2.2/1,
  ] ];
{
  return [ std_env, bridges, libs ];
}

A version selection model for a build which overrides bridges/c, bridges/cxx, and libs/c/z:

from /vesta/vestasys.org/platforms/linux/debian/i386 import
  // The version selection model from std_env
  std_env.sarge/1/vsel.ves;

// Use a newer version of the c_like bridge
from /vesta/vestasys.org/bridges import
  bridges = [ c = c_like/checkout/5/3,
              cxx = c_like/checkout/5/3 ];

// Use a newer version of zlib

from /vesta/vestasys.org/libs import
  libs = [ c/z = zlib/1.2.3/1 ];
{
  return std_env.sarge() ++ [ bridges, libs ];
}

A client model using this version selection model:

import
  // The above version selection model
  vsel = lix_i386.vsel.ves;
  // Local import
  self = build.ves;
{
  // Put the selected versions of each component into ./_models_ so that every function we call can get at them
  . = [ _models_ = vsel() ];
  // Preapre the build environment and merge it into dot.  (We're assuming it will leave ./_models_ as-is).
  . ++= ./_models_/std_env()/env_build();

  return self();
}

Problems with this approach:

  • It would be impossible to enforce strict version selection semantics. (SDL code could modify ./_models_, people could do non-local imports outside the version selection models, etc.) If we specifcally added version selection/overriding features to the evaluator we could better enforce rules (disable non-local imports, make the version selection namespace immutable once the evaluation starts, etc.)
  • Writing code to automatically edit the version selection models to add or remove overrides could be tricky. It would need to understand the difference between base configurations (called and combined with ++) from imports for overrides (placed in bindings and then combined with the base configurations).
  • We have to invoke the builder to get the current configuration. It would seem preferable to have a library for simply parsing a version selection file format.
  • The import clause syntax doesn't currently support creating nested bindings, or augmenting existing bindings with += or ++=. Those would be really handy for writing version selection files in general, and particularly for automating the editing of them.

Discussion