ARCJSCHMIDT.de
Thoughts from a developer//entrepreneur.
Founder of AETROS.com

open-source

php|Propel1 php|Propel2 php|CDS php|PHP-PM php|php-pm-httpkernel web|Jarves.io web|css-element-queries js|jQuery-selectBox php|PropelBundle php|Propel Sandbox js|angular2-localStorage js|angular-es6-annotations php|php-rest-service php|topsort.php xxx|BetterQuitJobBundle js|angular-typescript-decorators c++|node-core-audio php|optimistic-locking-behavior php|change-logger-behavior

github.com/marcj twitter.com/MarcJSchmidt plus.google.com/+MarcJSchmidt RSS xing.com/profile/MarcJ_Schmidt

Vagrant/continuous integration/dev machine with fixed third-party software/packages.

02 January 2014, by Marc

If you have a continuous integration server running or some kind of a dev/staging machine for your (software) release management you might considered already vagrant to ease that process. Especially, when you deal with VMs and software releases you usually want that each machine (dev, staging, live) have the exact same services/libs/package installed. Even better with automatically provisioning of your third-party dependencies/packages of that server.

To achieve that goal you have several ways. The hard way would be obviously to download all source packages (php/apache for example) and compile it on the initialized bootstrap of your VM. Beside the fact that this is a pretty time-consuming it’s also not that easy as you have to have all dependencies installed as well to get the compiling running.

The easier way is to install your packages and needed services via the integrated package manager of your (linux) distribution. I talk now here about Debian Package Manager (dpkg, which is also used Ubuntu).

Debian’s apt-get has the ability to talk not only to a remote repository but also to a local one. This gives us the possibility to store all installed *.deb files during the bootstrap in our own (git) repository and use those packages instead of talking to Debian’s official apt-get repositories. The advantage of storing all debian files to your repository is that you’ve then the ability to jump in the history of your server configuration. This is pretty cool, especially when you want to provision a default (dev) VM for all of your developers. Also a upgrade of some packages is then a ease as you only have to push a new debian package (.deb) to your git repository and all other can just pull one change instead of download a huge ISO again and again.

So, advantage of this method is:

Note: You might thought already about a local package version and installation with dpkg. This method has the big disadvantage that you have to resolve all dependencies on your own. That is very time consuming and frustrating, especially when you install a package with many small dependencies. You have to take care here about the installation order as well.

So, the only elegant way is to use the well known apt-get tool. The basic procedure is like this:

  1. Install all packages with a naked installation.
  2. Since all packages that have been installed through apt-get are cached in /var/cache/apt/archives/ you should move or copy all those packages to a own directory.
  3. Use a script (see below) to create a Packages.gz file.
  4. Update your /etc/apt/sources.list to point to your local repository.
  5. Next bootstrap is way faster and offline-ready.

Install all needed packages

In the initial step you have to install all packages in the regular way. Connect your machine to the internet and use apt-get install <packageName> as usual. If you machine is “dirty” (not a fresh installation), you have probably already some packages in /var/cache/apt/archives/. Make sure those packages a really required. Best way is to use a really fresh net installation. (like those of vagrant)

Create a own repository

In vagrant you have per default the /vagrant directory available. Since it’s a “mount” to your local filesystem to the directory where your Vagrantfile is located you’ve probably already a git repository there. If not, you should create a git repository and a sub-directory called ./apt-repository/.

Copy now all your *.deb files from /var/cache/apt/archives/ to ./apt-repository/ directory.

$ cp /var/cache/apt/archives/*.deb ./apt-repository/

To build the “index” file that is used in the debian repository to know which packages are available you can use following:

$ cd ./apt-repository/
$ dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Always when you add a new debian package (.deb) file to our ./apt-repository/ directory you have to run this command again.

Use our own repository

So, the process above is basically only needed in the very first step of the setup of your machine. You have now to say apt-get that it should use our own repository. Again, since in vagrant the /vagrant is available per default you don’t have to do something special to have all packages available.

Create at first a new sources.list file in your repository ./sources.list and add following content:

deb file:/vagrant/apt-repository ./

In your vagrant bootstrap (./bootstrap.sh) file:

(usually fired in ./Vagrantfile with config.vm.provision :shell, :path => "bootstrap.sh")

if [ ! -f /etc/apt/sources.list.old ]
then
    mv /etc/apt/sources.list /etc/apt/sources.list.old
    cp /vagrant/sources.list /etc/apt/sources.list
fi

apt-get update

Make sure this lines of bash code are placed at the very first. All following apt-get install calls use then our local repository.

The trick is now, each time you want to upgrade a package to a newer version you just have to move your source.list.old back and fire apt-get update again. Then just install the new packages as usual via apt-get install ..., move then all .deb files to your ./apt-repository and fire the dpkg-scanpackages from above again.

After committing and pushing your changes to the git repository all your developers or your continuous integration server is able to use automatically the newer version.

You can find more information about the debian repository structure in the manual Repository Howto.