- Repository Addresses
- Remote Checkout in Action
- Summary / Quick Reference
Vesta supports distributed operation. Once set up correctly, vcheckout can be used to transparently check out a package from a remote repository. While the user works on their change all operations happen locally. When they're done, vcheckin will replicate their changes back to the remote repository.
This page describes what you'll need to do to set this up.
Remote checkout is built on top of operations between peer repositories: replication and mastership transfer. Setting up for remote checkout is really a matter of setting up the access controls to allow these operations between particular repositories. After that, the repository tools take care of the details.
Experienced users may want to skip to the quick reference at the end.
Background Information Crash Course
It's assumed that you understand how hostnames, hostname resolution, and the basics of how TCP/IP works. It's assumed that you understand and have some control over the networks between the sites you want to set up. (Be sure to be aware of any firewalls or special packet handling such as NAT or VPNs.) Here's a few background resources on these topics:
Wikipedia has articles on:
The Linux Documentation Project has some instructional guides:
It's assumed that you understand what the term "realm" means in a Vesta context and how a realm is used to form global identifiers for users and groups. If not, see the repository(8) man page.
It's assumed that you understand how the basic Vesta operations like checkout and checkin work. Presumably you already have one Vesta server up and running and are familiar with its basic operation.
Here are some basic Vesta informational resources:
The vesta-intro(1) man page defines terminology and gives some basic background information.
It's assumed that you have secured the network connection between the peer sites. This document doesn't cover anything about encryption and relies upon simple host-based authentication.
Throughout this page examples will use a fictional company with two geographically distributed sites.
The fictional company's domain is "example.com".
The two sites are represented by the sub-domains "foo.example.com" and "bar.example.com". These sub-domains are also used as the realms for Vesta
The Vesta servers have the hostnames "vesta.foo.example.com" and "vesta.bar.example.com".
Client hosts (workstations used by individual contributors) have hostnames "client.foo.example.com" and "client.bar.example.com".
When not using a special account such as vadmin, we'll use the fictional user "jsmith". (In the two realms his global identities are of course "firstname.lastname@example.org" and "email@example.com".
Before we talk about the setup in detail, let's make sure the basic networking infrastructure is working correctly and that each site has simple access to the peer repository.
Hostnames, Addresses, Reachability
Each repository server must have a hostname which can be resolved to an IP address by its peer repositories. In our example, the servers are "vesta.foo.example.com" and "vesta.bar.example.com". We should start by logging into each of those and making sure that we can get an IP address for the other.
vesta.foo.example.com% host vesta.bar.example.com vesta.bar.example.com has address 10.2.1.1
vesta.bar.example.com% host vesta.foo.example.com vesta.foo.example.com has address 10.1.1.1
If you don't have the host(1) command try nslookup(1) or dig(1). Alternatively you can skip to the next step and let ping(8) do hostname resolution for you.
We should also make sure that each server can send packets to and receive from the other. One way we can do this is by using ping(8):
vesta.foo.example.com% ping vesta.bar.example.com PING vesta.bar.example.com (10.2.1.1) 56(84) bytes of data. 64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=1 ttl=64 time=0.232 ms 64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=2 ttl=64 time=0.225 ms 64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=3 ttl=64 time=0.225 ms 64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=4 ttl=64 time=0.232 ms --- vesta.bar.example.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 0.225/0.228/0.232/0.015 ms
vesta.bar.example.com% ping vesta.foo.example.com PING vesta.foo.example.com (10.1.1.1) 56(84) bytes of data. 64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=1 ttl=64 time=0.232 ms 64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=2 ttl=64 time=0.225 ms 64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=3 ttl=64 time=0.225 ms 64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=4 ttl=64 time=0.232 ms --- vesta.foo.example.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 0.225/0.228/0.232/0.015 ms
Note that ping may not work in all situations. Some firewalls will block ping but allow TCP connections through.
In some cases, it may be worth repeating the above tests from the client hosts as well ("client.foo.example.com" and "client.bar.example.com" in our example) to ensure that they can also resolve the hostname of and reach the repository at the other site.
In the examples on this page, we're going to use reverse hostname mappings (aka reverse DNS) to determine access. We should check that the reverse mappings are in place and give us the right hostnames.
vesta.foo.example.com% host 10.2.1.1 220.127.116.11.in-addr.arpa domain name pointer vesta.bar.example.com.
vesta.bar.example.com% host 10.1.1.1 18.104.22.168.in-addr.arpa domain name pointer vesta.foo.example.com.
You don't have to use reverse mappings for access controls. Alternatives will be mentioned.
Now that we've made sure have basic addressing and network connectivity, let's grant access to the peer sites. Each repository has an export file which controls which client hosts it will accept requests from. We'll need to add entries to the export file on each of our two repositories to allow the hosts at the other site access.
If you're using the installable RPM/Debian packages, you can find the export file in /etc/vesta/repos.export. If not, type "vgetconfig Repository export_file". If that's a relative path, it's below the directory printed by "vgetconfig Repository metadata_root".
In the export file on "vesta.foo.example.com" we'll add these lines:
# Allow any host in the bar.example.com access with the relam bar.example.com *.bar.example.com: allow global bar.example.com
Similarly, in the export file on "vesta.bar.example.com" we'll add these lines:
# Allow any host in the foo.example.com access with the relam foo.example.com *.foo.example.com: allow global foo.example.com
After modifying the export file, you can use the vaccessrefresh utility to get the running server to reload it. Alternatively, you can stop and restart the repository server process.
Instead of using reverse DNS, we could use a CIDR IP range. If all hosts in "foo.example.com" have IPs starting with 10.1 and all hosts in "bar.example.com" have IPs starting with 10.2 then we could have used "10.1.0.0/16" and "10.2.0.0/16" instead.
Now that we've granted some access, let's check that the remote repository is allowing us in. We can do this with the vid utility.
vesta.foo.example.com% vid -R vesta.bar.example.com User names and aliases: firstname.lastname@example.org Groups: Unix (NFS) user ID: 1002 Unix (NFS) primary group ID: 1002
vesta.bar.example.com% vid -R vesta.foo.example.com User names and aliases: email@example.com Groups: Unix (NFS) user ID: 1002 Unix (NFS) primary group ID: 1002
Each repository has a public address which peer sites use to contact it. This is a hostname and a port and is written as a string: "host:port". Assuming we're using the default ports, our two example repositories addresses are:
Specifying Your Public Address
Ideally all repository clients will use the public address to access the repository. If this is the case, the repository's public address will simply be taken from the config file settings giving the repository's host and port.
However, this may not be feasible in all cases. For example, a laptop installtion will want to use "localhost" to access the repository yet may still have a public address which is valid on a certain network. In cases like this, you can set the configuration variable [Repository]master_hint to explicitly specify the public address of your repository.
When you create a new master appendable locally (e.g. a new top-level directory created by vwizard), the repository automatically stores the public address of your repository in a "master-repository" attribute on that object. This attribute can be propagated (along with the object) to a replica.
Whenever you transfer mastership of an object between repositories, "master-repository" attributes get added/updated.
This is why it's important to get your repository's public address set correctly (e.g. with the [Repository]master_hint configuration setting). If the wrong address gets recorded, or if you don't have any "master-repository" attributes, you may need to set them explicitly.
The repository won't set any master-repository attributes if you haven't set [Repository]master_hint and your repository's address is "localhost". Obviously "localhost:21776" won't be usable for peer repositories to contact your repository.
Suppose for example that the top-level directory /vesta/foo.example.com was created in the repository at vesta.foo.example.com. Maybe the initial setup didn't use a fully-qualified hostname (and [Repository]master_hint wasn't set either). The master-repository attribute would have recorded this incomplete address:
vesta.foo.example.com% vattrib -g master-repository /vesta/foo.example.com vesta:21776
We can simply correct this with vattrib:
vesta.foo.example.com% vattrib -s master-repository vesta.foo.example.com:21776 /vesta/foo.example.com vesta.foo.example.com% vattrib -g master-repository /vesta/foo.example.com vesta.foo.example.com:21776
If /vesta/foo.example.com has already been replicated to the peer repository at vesta.bar.example.com, the attribute will be wrong there as well:
vesta.bar.example.com% vattrib -g master-repository /vesta/foo.example.com vesta:21776
vesta.bar.example.com% vrepl -v -s vesta.foo.example.com -e+ /vesta/foo.example.com -e- "/vesta/foo.example.com/*" exists /vesta/foo.example.com attribs /vesta/foo.example.com
Of course this assumes that you've already set up everything correctly for replication, which we'll talk about in the next section.
Use For Access Control
There are three special access control attributes which are used during distributed operations:
#replicate-from lists the repositories to be trusted as replication sources
#mastership-to lists the repositories to which we're willing transfer masterhip
#mastership-from lists the repositories to from which we're willing accept a transfer of masterhip
The values of each of these attributes are repository addresses. Each will be described in more detail below.
Replication is the process by which objects under /vesta are propagated from one repository to another.
There's a short overview of replication on the vrepl(1) man page. If you want to understand it in depth, see "Partial Replication in the Vesta Software Repository" (which also explains mastership, which we'll talk about a little later).
Several access control checks are performed in the course of a replication:
- The client host accesses both repositories to find the objects to be replicated. For this, the client host and the user running the tool must have read access to both repositories.
- The client calls the destination repository asking it to perform a replication from the source repository. To perform a replication, as user must have read/write access to the destination repository.
The destination repository checks that the source repository is trusted as a replication source. This is done by checking that the source repository is listed in a #replicate-from attribute at the destination.
The destination repository reads the contents of the replicated objects from the source repository as the requesting user. For this, the destination repository must be granted access to the source repository with the realm of the requesting user.
To support remote checkout/checkin, we'll need to allow bi-directional replication in both push and pull modes. The local repository (where the checkout is performed) will need to replicate objects from the remote repository. When checking in, the local repository must replicate the new version back.
We'll go through each of the access checks listed above and make sure that we've set up the access that we need.
Finding Objects to Replicate
The first step in performing a replication is checking both the source and destination repositories to find objects which need to be replicated. Sometimes a replication may specify a complex pattern to be matched (e.g. using the replicator's directive syntax). Also, the object(s) might already be present at the destination.
To do this we need read access to both repositories from any client host that might perform a replication. In our example, the remote site is bar.example.com. We need to make sure that users at that site can access the objects we're interested in replicating in both repositories. Earlier we checked that the repository server would allow them to make requests with vid, but that doesn't guarantee that they will have sufficient permission to access packages and versions at the repository on vesta.foo.example.com.
We'll assume that there's a package named /vesta/foo.example.com/a which contains a version 1. Let's eaxmine its attributes from the remote site:
client.bar.example.com% vattrib -R vesta.foo.example.com /vesta/foo.example.com/a/1 vattrib: /vesta/foo.example.com/a/1: No permission
We'll need to solve this before we can replicate. There are several possible ways we could solve this:
Turn on world read and execute permissions for the enclosing directories (i.e. "chmod o+rx /vesta/foo.example.com/a /vesta/foo.example.com /vesta" over at foo.example.com)
Add the requesting user (firstname.lastname@example.org) to a group which has read permission for this package in the repository group file on vesta.foo.example.com
For example, you could use this line: email@example.com: ^firstname.lastname@example.org
Note that, as with changing the export file, you'll need to use vaccessrefresh or restart the repository.
Give the requesting user permission to act as a local user which has read permission for this package in the repository alias file on vesta.foo.example.com
For example, you could use this line: email@example.com: firstname.lastname@example.org
- Again, you'll have to force the repository to reload its access control information
After doing at least one of these, our user should have permission to access the package and its version, so let's try our test again:
client.bar.example.com% vattrib -R vesta.foo.example.com /vesta/foo.example.com/a/1 master immutableDirectory message initial version of package a content /vesta/foo.example.com/a/checkout/1/5 checkin-time Mon Oct 29 14:33:10 EST 2005 checkin-by email@example.com #owner firstname.lastname@example.org
We should also repeat this test in the opposite direction, having a user at foo.example.com try to access objects in the repository at vesta.bar.example.com (similarly modifying access control information to grant access). In our example we haven't yet replicated anything to vesta.bar.example.com for us to access yet. However, it would be a good idea to revisit this later.
Requesting the Replication
Earlier, we granted each site full access to the peer repository. (That is, in the repository export file we used "allow global" rather than "readonly global".) This should allow us to perform replications in either direction.
Trusting the Source Repository
Each repository can control which other repositories it trusts as replication souces. Vesta relies on the full path to an immutable version always referring to identical contents in all repositories. However, this is not strictly enforced, so you should only choose to allow replication from repositories which you are reasonably certain you can trust.
The #replicate-from attribute lists the addresses of repositories to be trusted as replication sources. This can be placed on /vesta or subdirectories of it. In order to determine whther a particular replication is allowed, the repository searches upward from the path to be replicated for the closest enclosing directory with a #replicate-from attribute. The source repository's address must be listed in that attribute or permission for the replication will be denied.
In our example, we first need to allow replication from the master repository (vesta.foo.example.com) to the remote site (vesta.bar.example.com). So we'll add a #replicate-from attribute which allows this:
client.bar.example.com% vattrib -a #replicate-from vesta.foo.example.com:21776 /vesta
You'll need write permission to /vesta in order to add this attribute, which probably means using the special vadmin user or root.
Now we'll try a replication of the version we inspected above (/vesta/foo.example.com/a/1):
client.bar.example.com% vrepl -v -s vesta.foo.example.com -e+ /vesta/foo.example.com/a/1 create /vesta/foo.example.com create /vesta/foo.example.com/a copy /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1 latest /vesta/foo.example.com/a/latest attribs /vesta/foo.example.com/a attribs /vesta/foo.example.com
Now we're all set to replicate from the remote master. Hwoever, we also need to be able to replicate back to the remote master so that we we check in a new version from bar.example.com it will appear in the repository at vesta.foo.example.com. If we try to replicate the same version back, we'll run into trouble:
client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1 exists /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1 vrepl: No permission, copying attribs for /vesta/foo.example.com/a/1
This is because we haven't yet granted permission to replicate in the other direction. We'll have to add a #replicate-from over in the repostiory at vesta.foo.example.com.
vesta.foo.example.com% vattrib -a #replicate-from vesta.bar.example.com:21776 /vesta
Again, you'll probably need administrator access to do this.
Now we should be able to perform the replication in the other direction:
client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1 exists /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1
Just to convince ourselves that this is working, let's make an attribute change locally and make sure that it gets propagated:
client.bar.example.com% vattrib -a "test-attribute" "jsmith@bar was here" /vesta/foo.example.com/a/1
client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1 exists /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1 client.foo.example.com% vattrib -g "test-attribute" /vesta/foo.example.com/a/1 jsmith@bar was here
Access from Destination as Requestor
When performing a replication, the destination makes requests to the source repository to read the objects being replicated. These requests are made with the identity of the user requesting the replication.
When the local repository is the destination and the source is a remote rpeository, this seems perfectly natural. For example, above the user ran the replicator on client.bar.example.com to copy an object from the remote repository vesta.foo.example.com to the local repository. The requests from the destination repository host used the global identity of the user running the replicator (email@example.com). This matches the line we added to the repository export file earlier.
This can be thought of as a "pull" replication.
If the local repository is the replication source and the destination is a remote repository, things are a little different. If the user ran the replicator on client.bar.example.com to copy an object from the local repository to the remote repository, requests would arrive from vesta.foo.example.com with the global identity of the user running the rpelicator (firstname.lastname@example.org). This would not match the export file entries we added earlier.
This can be thought of as a "push" replication.
To support replication in from the local repository to a remote repository, we'll need to add some entries to the export files of our two repositories. In the export file on "vesta.foo.example.com" let's add these lines:
# Allow peer repository in with our realm to support "push" replication vesta.bar.example.com: allow global foo.example.com
Similarly, in the export file on "vesta.bar.example.com" we'll add these lines:
# Allow peer repository in with our realm to support "push" replication vesta.foo.example.com: allow global bar.example.com
Remember that after modifying the export file, you need to get the repository server to reload it either with the vaccessrefresh utility or by stopping and restarting the repository server process.
Now let's test this:
client.foo.example.com% vrepl -v -d vesta.bar.example.com -e+ /vesta/foo.example.com/a/1 exists /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1
client.bar.example.com% vrepl -v -d vesta.foo.example.com -e+ /vesta/foo.example.com/a/1 exists /vesta/foo.example.com/a/1 attribs /vesta/foo.example.com/a/1
Each object under /vesta has a master flag. You can check the master flag with vattrib which prints "master" or "nonmaster".
Only a single repository can have the master flag set for an object at once. That is the master repository for that object. The replica of an object at its master repository is called the master copy of the object. Changes to an object can only be made at the master repository. For example, if you want to check out a package, its master repository must grant you permission to do so.
Finding the Master
The repository tools locate the master copy of an object using several methods:
Through master-repository attributes. These contain a pointer to the peer repository which is believed to have mastership of the object. (These are only considered hints and are not necessarily authoritative.)
Additional repositories to be checked can be specified in the configuration file with the setting [UserInterface]DefaultHints.
Some tools allow the user to specify additional repositories to check on the command line.
Mastership transfer is the process by which the master flag can be moved from one repository to another. vmaster can be used to explicitly transfer mastership, but it's much more common for tools like vcheckout to automatically transfer mastership for the user.
The #mastership-to attribute lists the addresses of repositories we're willing transfer masterhip to.
The #mastership-from attribute lists the addresses of repositories we're willing to accept a mastership transfer from.
You can probably skip this description of the details of how mastership transfer works.
Mastership transfer is a handshake protocol. It proceeds in several steps:
- The client locates the master repository.
Let's suppose that the master is vesta.foo.example.com.
The client contacts the repository it wants to become the master and asks it to acquire mastership from the current master.
Let's suppose the client wants vesta.bar.example.com to become the master. It tells vesta.bar.example.com "acquire mastership from vesta.foo.example.com:21776"
After deciding whether it will accept this transfer, the repository acquiring mastership contacts the current master and asks it to cede mastership to it.
In our example, the repository at vesta.bar.example.com contacts the repository at vesta.foo.example.com and asks "please cede mastership to vesta.bar.example.com:21776".
- Note that the request to cede mastership goes from the destination to the current master with the identity of the user requesting the transfer. (This is similar to the way replication works.) Normally this is a user local to that repository, so the export file at the current master probably already allows this.
- After deciding whether it will accept this transfer, the repository ceding mastership turns off the master flag and contacts the new master and and tells it that it has given up mastership.
In our example, the repository at vesta.foo.example.com contacts the repository vesta.bar.example.com and says "I've given up mastership, you can have it now"
Note that this also uses the identity of the requesting user. This user probably isn't local to the repository making the request, so you'll need to allow their realm in from the peer repository. In our example, this means that the export file on vesta.bar.example.com must allow vesta.foo.example.com with realm bar.example.com. (This is similar to what we need for "push" replication.)
- Finally, the repository acquiring mastership turns on its master flag and indicates to the client that it has acquired mastership.
Mastership transfer is designed to maintain the special property that at most one repository has the master flag set even if something bad happens during the transfer such as a loss of network connectivity or one or both repository servers going down (e.g. from a power loss). The repository server will automatically retry partially completed mastership transfers.
Normally this is completely transparent and users never have to be aware of it. However, it's worth mentioning because if a repository permanently goes off line peer repositories may continue trying to contact it.
Remote Checkout in Action
Now that we've got all the necessary permissions in place, let's perform an actual remote checkout. Our user jsmith simply uses vcheckout the same way they would for a package that originated in the local repository:
client.bar.example.com% vcheckout /vesta/foo.example.com/a Reserving version /vesta/foo.example.com/a/2 at vesta.foo.example.com:21776 Creating session /vesta/foo.example.com/a/checkout/2 at vesta.foo.example.com:21776 Making working directory /vesta-work/jsmith/a
The only difference visible to the user is the "at vesta.foo.example.com:21776". That tells the user that a remote repository was contacted for those parts of the operation. After creating the version reservation and the session directory at the remote master, vcheckout automatically replicates them to and transfers mastership of them to the local repository. We can see that the local repository now has mastership of the reservation with vattrib:
client.bar.example.com% vattrib /vesta/foo.example.com/a/2 master stub work-dir /vesta-work/jsmith/a session-dir /vesta/foo.example.com/a/checkout/2 old-version /vesta/foo.example.com/a/1 master-repository vesta.bar.example.com:21776 checkout-time Fri Dec 2 13:27:46 EDT 2005 checkout-by email@example.com #mode 775 checkout-to vesta.bar.example.com:21776 checkout-from vesta.foo.example.com:21776
Also worth noting are the last two attributes:
checkout-to records the address of the local repository where the change is being made
checkout-from records the address of the remote repository which has mastership of the package
These attributes are also used by other tools. For example, a user at foo.example.com running vwhohas will see that a user at bar.example.com has the package checked out:
client.foo.example.com% vwhohas /vesta/foo.example.com /vesta/foo.example.com/a/2 firstname.lastname@example.org vesta.bar.example.com:21776
All the user's activities before checking in happen locally. Editing and taking snapshots with vadvance only use the local repository. Even filling in the reserved version on checkin happens locally. After doing the local part, vcheckin will replicate the version back to the repository in the checkout-from attribute:
client.bar.example.com% vcheckin /vesta-work/jsmith/a Checking in /vesta/foo.example.com/a/2 Deleting /vesta-work/jsmith/a Replicating /vesta/foo.example.com/a/2 to vesta.foo.example.com:21776
This replication isn't strictly necessary, and if it fails the checkin will still be complete locally. The replication could be re-done later with vrepl. However, /vesta/foo.example.com/a/2 would still be a reservation stub rather than the checked-in version over at vesta.foo.example.com.
Summary / Quick Reference
Skipping over all the detailed exaplantion and examples, here's everything you need to have set up for remote checkout to work:
- You must have a functional TCP/IP infrastructure with connectivity between the peer sites.
- Each repository must have a hostname which resolves to an address which can be used to contact that repository at any site.
- In other words, the hostname of your repository must be usable for contacting it from a remote repository and vice versa.
Users must be granted read/write access to all participating repositories through the export file
In other words, use "allow global" not "readonly global"
- Read-only access is sufficient for a replication source, but not mastership transfer
- Each peer repository must also be allowed in with the local realm to allow transactions initiated by local users which pass through remote repositories
- This is needed for both mastership transfer and "push" replication
vcheckin performs a "push" replication to copy the new version back to the remote master
- This is not needed for strictly "pull" replication
- This is needed for both mastership transfer and "push" replication
All peer repositories must have their addresses listed in a #replicate-from attribute which applies to the hierarchy in which you'll be performing remote checkouts
A remote checkout replicates from the master repository
A remote checkin replicates to the master repository
The master repositories of packages must have their addresses listed in a #mastership-from attribute which applies to the hierarchy in which you'll be performing remote checkouts
- The repository local to the checkout must be willing to accept a transfer of mastership for the new version reservation stub and the session directory
It's advisable to treat this like #replicate-from and allow transfers in either direction between all peers
The non-master repositories which will be local to checkouts must have their addresses listed in a #mastership-to attribute which applies to the hierarchy in which you'll be performing remote checkouts
- The master repository of the package must be willing to grant a transfer of mastership for the new version reservation stub and the session directory to the repository local to the checkout
It's advisable to treat this like #replicate-from and allow transfers in either direction between all peers