Camp::Master - library routines for management of camps
3.03
The camp system relies on a single master database to hold the known camp users, camp types, and existing camps. All camps within the system are registered within this database, along with the type of camp, the version control system used for that camp, the owner, etc. The camp system will not function without this database.
Connectivity info for this database must be provided to the camp system. The camp master database configuration file, camp-db-config, serves this purpose.
It is expected to live at $camp_base/camp-db-config.
Entries consist of the form $key:$value, one per line.
The following key/value pairs are expected to be provided by this file:
The DBI DSN for the master camp database. See the DBI docs for DBI well-formedness; it depends in no small part on the database driver used (currently, only Pg and mysql drivers are supported).
The username to use when connecting to the master camp database.
The password to use when connecting to the master camp database. If your database is configured to allow access without a password, then you may leave this option out; however, that is typically not recommended, since the connection attempts will happen from different UNIX user accounts.
Therefore, a typical camp-db-config file might look like:
dsn:dbi:Pg:dbname=camps user:camp password:fuggedabowwditt
Your camp type may specify arbitrary paths that should be copied/symlinked into camp instances via the copy-paths.yml file. As the extension suggests, this is expected to be a YAML file.
The structure of the file is as follows:
Each path to copy (file or directory, it matters not) gets a "document" within the YAML structure. This essentially means that you start each path's entry with the standard "---" YAML document header. The result of this is that Camp::Master sees the path entries as an array.
Each path entry is a hash. Each path hash must provide the following key/value pairs:
The source path to copy from. This can be relative to the camp type path, or absolute.
The target path to copy to. This is always relative to the camp instance base.
In addition, the following optional items may be provided:
If provided with a Perly true value (non-blank, non-zero), the target will be created as a symlink to source rather than a full copy. This is sensible for large docroots, image directories, etc.
If left unspecified for a given path, this effectively defaults to false.
If provided with a Perly true value (non-blank, non-zero), the default behavior determined by default_link is always used and the camp creator isn't given a choice in the matter.
May be provided with a single scalar entry or an array of such entries; each should correspond to an rsync/tar-style "--exclude" pattern, which will be provided to the rsync call that performs the copy. This naturally only comes into play if copying the source rather than symlinking.
If provided with a Perly true value, the source path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.
If provided with a Perly true value, the target path will be parsed for camp configuration tokens (see CAMP CONFIGURATION VARIABLES). If not specified, parsing does not occur.
Because the YAML stream is converted to an array, the paths are processed in order of specification within the stream. Therefore, you can have dependencies between paths and they'll work out as long as you arrange them in the proper order in the file.
When a new camp instance is created, each path is processed in order. Per path, the camp creator is offered the opportunity to specify whether the camp should receive a full copy or a symlink of the source path in question; the default response will be noted based on the default_link setting for that path. However, if the always option is set for the path, the user is not given the choice and the default simple happens automatically.
Here's an example of a file:
# The /var/www/html docroot will be symlinked to $camp/htdocs by default, # but the user is given the choice to override with a copy. --- source: /var/www/html target: htdocs default_link: 1 # The /var/cgi-bin/foo.com script directory will by copied (since no default_link # is specified) by default to $camp/cgi-bin. The user is given the choice to # symlink instead. --- source: /var/cgi-bin/foo.com target: cgi-bin # The /var/www/image_repository is symlinked to $camp/image_repository; the user # is not given a choice about this, because always is set. --- source: /var/www/image_repository target: image_repository default_link: 1 always: 1
Camp configuration occurs at two levels:
Configuration information provided at the $camp_base trickles down to all camp types underneath that base.
Each camp type has a subdirectory under the $camp_base, and configuration information provided at this level applies only to the camp type in question, overriding any conflicting settings coming in from the base camp level.
The configuration information is parsed at the base camp level first, then the type-specific level second.
In each level, the local-config file is processed. This is expected to consist of key/value pairs of the form:
some_key:some_value
Empty lines and lines starting with the pound character are ignored, meaning that you can embed comments in your configuration files.
All values specified in these files undergo token substitution, such that any configuration variable can be substituted into a value by using the token:
__CAMP_VARIABLENAME__
For instance, if the local-config file has a line:
my_variable:some_useless_value
then a subsequent line:
other_variable:__CAMP_MY_VARIABLE__foo
will result in the "my_variable" key having value "some_useless_valuefoo".
These variables are used both by Camp::Master itself, and in the various templates that Camp::Master renders at camp creation time; the token substitution logic is the same in each case. This allows your template files to undergo rendering and have camp-specific information substituted in, localing the file to the camp being created (which is the entire point of the camp system).
Prior to parsing the base level and camp level configuration files, a number of configuration entries are determined automatically by Camp::Master.
The $camp_base (defaults to /home/camp)
The path to the camp type's directory under $camp_base.
The type of camp.
The camp owner's home directory path.
The local path of the camp being operated upon (for instance, /home/some_user/camp16)
The camp number
The camp's name (for instance, "camp23")
The Apache docroot for the camp.
The custom port the camp will use for HTTP traffic.
The custom port the camp will use for HTTPS traffic.
The camp-specific directory where Apache configuration and log files will go.
The path to the Apache modules that must be referenced when launching camp Apache.
Defaults to /usr/lib/httpd.
The path to the Apache executable for controlling the Apache server.
Defaults to /usr/sbin/httpd.
A space-separated list of parameter names that should be defined for Apache at startup time (via -D on the command line). These can be tested for in Apache configuration with the IfDefine parameter. See http://httpd.apache.org/docs/2.2/mod/core.html#ifdefine for details.
Specify location of configuration file under "httpd_path" and include that in command strings.
For example, "conf/apache2.conf", used when executable has been compiled with a specific full path. To detect need for this option see httpd -V output and check for "SERVER_CONFIG_FILE" with a value similar to "/etc/apache2/apache2.conf" (any path beginning with "/").
A boolean (0/1) setting that determines whether to skip Apache-specific httpd server setup, for example when using nginx.
When setting skip_apache, set this to the command to start your webserver. An example with nginx:
/usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
When setting skip_apache, set this to the command to stop your webserver. An example with nginx:
pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill $pid
pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill -HUP $pid || /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
A boolean (0/1) setting that determines whether to skip generation of a self-signed SSL certificate for the HTTP server.
Default is 0 (false).
The main interchange directory within the camp.
The path where the interchange linker programs should reside; defaults to cgi-bin under the camp's path.
The main path for your Interchange catalog; requires that the catalog variable be provided in your configuration file.
Defaults to path + '/catalogs/' + catalog.
The path where rails will live within the camp; defaults to "rails" under the camp's path.
The lowest port to be used by the Mongrel cluster for balancing Rails children
The ProxyBalancer name used for the Mongrel listeners within the camp Apache.
Rendered Apache ProxyBalancer member configuration directives suitable for placement directly within a virtualhost's <Proxy balancer://...>...</Proxy> container. Defaults to using three mongrel listeners with ports incremented by one starting at the mongrel_base_port.
When using an application server other than those natively supported in camps, set this to the command to start your application server. For example (using a custom startup script):
__CAMP_PATH__/bin/start-unicorn
Command to stop your application server. An example for Ruby's Unicorn:
pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid
Command to restart your application server. An example for Ruby's Unicorn:
pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill -HUP $pid || __CAMP_PATH__/bin/start-unicorn
The type of database server used for the camp; will be either 'pg' for Postgres or 'mysql' for MySQL.
The database type is determined by the subdirectories within the camp type path; the camp type path must have either a pgsql directory or a mysql directory (for pg or mysql, respectively).
The main database path for the camp's database server; dependent on the database type, will be either 'pgsql' or 'mysql' for Postgres and MySQL respectively.
The database hostname; defaults to 'localhost'.
The custom port to use for your camp's database server. Defaults to 8900 + camp number.
The encoding to use when initializing the database cluster. Defaults to UTF-8.
The directory where the camp's database is expected to store binary data. Defaults to db_path + '/data'.
A temporary directory for varying, ephemeral data like logs, pids, etc., specific for the camp's database server. Defaults to db_path + '/tmp'.
The base database server logfile path; for Postgres, this is in fact the logfile to be used; MySQL uses multiple logging paths ordinarily, so it can be used as a prefix for a full path with MySQL (it probably could be used for all logging too).
Defaults:
db_tmpdir + 'postgresql.log'
db_tmpdir + 'mysql.log'
The database server configuration file within the camp.
Defaults:
db_data + 'postgresql.conf'
db_path + 'my.cnf'
Path for the camp's database server UNIX socket. Defaults to db_tmpdir + 'mysql.$camp_number.sock'
All of the above variables are calculated for you. This calculation is one of the first things performed by Camp::Master; therefore, they can be safely overridden in your base or type configuration files as appropriate for your camp deployment.
A few variables must be provided by your configuration files in order for the camp system to function properly:
Space-separated list of paths to SQL files that should be run upon preparing a camp's database server. You may specify as many as you need, but at least one should be provided.
The paths may be absolute or relative; if relative, they are expected to be relative to the camp type's directory.
Each script specified in the space-separated list is run within its own db client shell; they are executed in order of specification. No assumptions are made for transaction handling; if you want transactions, put them in the scripts themselves.
It's important to know the context in which these run, which differs between database server types due to fundamental structural differences between those servers and how they organize things:
In Postgres, each script is run via a connection to the "postgres" user's "postgres" database. Therefore, your scripts need to provide \connect information in order to make changes against a different database.
In MySQL, the scripts connect using the db_default_database and db_default_user variables, which effectively makes these variables required for MySQL deployments.
Space-separated list of paths to SQLfiles that should be run upon preparing a camp's MySQL database server, prior to processing the roles for that database; this gives an opportunity to initialize the database to a known set of users/permissions, for instance, or initalize things in other ways needed prior to role creation.
Allows specification of a number of seconds to sleep between starting the database server at camp creation time and processing the roles and db_source_scripts; necessary for UNIX socket connections in which the server needs a few seconds to start up and initialize its socket file.
Sane values appear to be in the 5-15 second range. Defaults to 5 seconds; set to 0 or blank to turn off.
Space-separated list of database names that are expected to live within your camp's database server.
It's important to provide this list, particularly for Postgres, so your user account's .pgpass file can be updated appropriately to allow easy access to each database specified. Ordinarily, only one database would be listed.
A space-separated list of directories that are expected to reside under your camp (naturally, these paths are expected to be relative to the camp directory itself). These directories will be cleared out if you rebuild an existing camp (with the mkcamp script), which is the main reason they need to be provided to Camp::Master.
Space-separated list of filenames that need to be put in place within your camp as Apache/Interchange linker programs. Specify at least one for your main catalog.
The name of the catalog for your camp's Interchange deployment. Specify one, even if you have multiple catalogs; the catroot variable is set based on this. For the majority of End Point Interchange deployments, multiple catalogs are a non-issue.
=back
Some optional variables may also be provided to affect the functioning of your camp deployment:
The locale to use when initializing the database cluster. There is no default, meaning that the cluster would by default initialize to whatever locale Postgres itself uses as the default.
The default database to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the database used for db_source_scripts import on MySQL.
The default user to use when connecting to your database via its respective camp DB client wrapper (psql_camp or mysql_camp). Also determines the user used for db_source_scripts import on MySQL.
The country to use for your camp's SSL certificate (defaults to US).
The statename to use for your camp's SSL certificate (defaults to 'New York').
The locality to use for your camp's SSL certificate (defaults to 'New York').
The organization name to use for your camp's SSL certificate (defaults to 'End Point Corporation').
Specifies the default version control system to use for the camp type. This is optional, but is certainly highly convenient for camp users and makes things easier to manage, as providing this means that users don't typically need to question whether they should be using a Subversion-based VCS type ('svn' or 'svk') versus a Git type ('git').
If this is set for a camp type, then it is not necessary for a user to specify their VCS type when creating an instance of that camp. The value of vcs_default_type has no bearing on existing camps; each extant camp's original VCS type is recorded in the camp database and will not change without heroic efforts on the part of the camp's owner.
Use this to specify the SCM repository path. This is not specific to VCS type and has been preserved for legacy purposes. SVN and Git repository paths can be specified in type-specific manner using repo_path_svn and repo_path_git, respectively; when those are used, they override the repo_path value.
There is no default value for repo_path, so don't count on it being present in your code unless you are quite sure of what you're doing.
Use this to specify the main project Subversion repository path; is is assumed that it will be accessed via the "file://" protocol, meaning that you need only provide the path to the repository (and any subdirectories within that repository).
When determining the Subversion repository location, Camp::Master consults, in this order:
repo_path_svn
repo_path
Default: type_path . '/svnrepo/trunk'
There is no default value for repo_path_svn, however, so do not count on this being available for token substitution unless you explicitly set it in the relevant local-config. The default repository location described above is enforced through logic, but not within the master configuration hash.
Use this to specify the main project Git repository path; no assumptions about the path/URL are made, so you are free to specify a file path (which is most likely as of this writing), a git://* URL, an rsync://* URL, etc.
Similar to determination of Subversion repository location, Camp::Master consults the following items in order when finding a camp type's Git repository location:
repo_path_git
repo_path
Default: type_path . '/repo.git'
There is no default value for repo_path_git, so you cannot count on its presence unless you know what you're doing and are working with a camp type that definitely specifies a value for it.
Specifies the SVK mirror path used for mirroring the master SVN repository. The convention is for the mirror and working paths to live under the default SVK depot, meaning that they should begin with '//' rather than '/some_depot_name/'.
If not specified, the SVK mirroring path used will be:
'//mirror/' . I<type> . '/trunk'
Like repo_path, there is no default value for this particular setting.
Options to pass literally to the "git clone ..." call when building a new camp and cloning from the source Git repo. See the "git-clone" man pages for valid options. There is no default, meaning that the effective default behavior is invoking git-clone without options.
Git is efficient in its use of disk space, but by using the various options like local, shared, or reference can potentially be even more efficient and speedy.
Specifies the SVK local working path used per camp; when creating a new camp using the 'svk' version control system option, a branch is always created from the main mirror path (see repo_mirror) to this local path. The local path is assumed to be specific per camp.
If not specified, the SVK local working path will be:
'//local/' . I<type> . '/' . I<name>
Like repo_path, there is no default value for this particular setting.
Rendering and installing arbitrary files into a camp from templates is a critical aspect of the camp system. Configuration files for Apache, Interchange, Rails, etc. should be reduced to templates, with things like hostnames, file paths, port numbers, etc. replaced with camp-system tokens of the form described in the section regarding CONFIGURATION VARIABLES. At camp creation time, Camp::Master will parse each template, performing token substitution, and install each parsed template into the appropriate location within the new camp.
The files to parse are specified within the camp type subdirectory's camp-config-files file. Specify one path per line; blanks and lines starting with the pound character are ignored.
All the files specified will undergo the described token substitution and installation into the new camp. The paths specified in camp-config-files are expected to be relative to the camp type directory's 'etc/' subdirectory, and also reflect the target path of the file when copied into the new camp itself. Thus, a file at /home/camp/some_type/etc/blah/foo.conf would be registered in camp-config-files with a path relative to etc/, or "blah/foo.conf", and would be installed at /home/some_user/campNN/blah/foo.conf after parsing.
Also, some assumptions are made about what files will be included if your camp uses Interchange versus Rails:
If your camp-config-files file specifies interchange/bin/interchange, then it is assumed to use Interchange.
If your camp-config-files file specifies rails/.../config/mongrel_cluster.yml, then it is assumed to use Rails.
Note that these are not mutually exclusive. It is theoretically acceptable for a single camp to employ both app server types. If you are bursting with curiosity and sadly deficient in sanity, by all means try this out.
While this can be expected to vary between deployments, a number of files are obvious candidates for inclusion within this templating scheme:
rails/.../config/database.yml
interchange/interchange_local.cfg
catalogs/.../catalog_local.cfg
httpd/conf/sites/some_site.conf
Anything that relies on file paths, ports, domain names, etc. should be abstracted out into such templates. A common pattern (implied by interchange_local.cfg and catalog_local.cfg above) is to encapsulate such details within "local" configuration files that are included from master configuration files; the master configuration files can stay in version control and have no need to vary between camps, production, etc., while the "local" configuration files are small, containing only what is absolutely necessary, and vary in a controlled way between environments by virtue of being managed by the camp system.
The camp system provides three different version control choices:
Subversion ('svn')
SVK ('svk')
Git ('git')
These can be further categorized into two options: Subversion-based ('svn' and 'svk') versus Git. The SVK version control system uses an alternate interface for working with the repository, setting up the camp, etc., but it relies on an underlying Subversion repository and naturally will not work unless the camp type has a Subversion repository available.
Typically speaking, use of Git for a particular camp type implies non-use of Subversion, and the opposite also holds. However, there is no reason that all types cannot be made available for use within the same camp type; the Git software suite comes with '''git-svn''', which can be used to track a Subversion repository via a Git repository. Therefore, all options can be available for a camp type provided the administrator takes the trouble to arrange it. That arrangement takes place largely outside of the camp system itself, and would instead involve post-commit hooks within one or both repositories to ensure that a change in one repository is pushed to the other.
Camp::Master ultimately makes no assumptions about your camp type's VCS offerings; if a camp user attempts to make a new camp using some version control system, Camp::Master will attempt to do it, and if the requested version control system hasn't been configured for the camp type, an error will result. Configuration options to tell Camp::Master about your VCS setup are fairly straightforward; see the configuration variable information for:
default_vcs_type
repo_path
repo_path_git
repo_path_svn
repo_mirror
repo_svk_local
While no assumptions are made, the strong recommendation as of this writing is, given all options, use Git. Subversion is slow, inefficient in its use of disk space, relatively inflexible, bad at branching, etc. The SVK system can be used on top of Subversion to accomodate for some of this but it ultimately is, in the words of Jon Jensen, "weak sauce". Git does everything that SVN+SVK do and far more, more efficiently, more elegantly.
When creating a new camp using Git as the VCS type, Camp::Master assumes an SVN-like flow of operations; the camp type specifies where the Git repository lives, and the new camp will result in a new repository that tracks that central repository. Consequently, the operations used by Camp::Master tend to be clone, pull, and push operations, for remote repository use. When you're working within your Git-based camp, you of course need to keep this in mind, knowing that a checkout or a commit are only relative to the repository of that camp, and that you ultimately need a push to commit your stuff to the central camp repository.
At this time, Camp::Master doesn't provide any tools for creating a Git camp using an alternate repository as the repository to track; this is something that can be done using git directly, after creating a new camp. Create the camp with Git as the VCS choice, then use git directly within that camp to point the camp at an alternate remote repository.
The camp system is designed to support easy use of SVK, which effectively allows for easier branch management and more effective management of camps that are undergoing major revisions over long periods of time (and thus less likely to be committed to the main repository on a frequent basis).
In order for SVK to work on your deployment, you must have svk installed (which is an exciting adventure, placing various requirements on your SVN version, for example).
There is no need to configure SVK per user; the camp system will initialize a user's local .svk settings, mirrors, etc., whenever that user first chooses 'svk' as their version control system for a new camp.
If you want to make use of SVK with the camp system, please look at the configuration variables section, with specific attention paid to repo_mirror and repo_svk_path; these dictate how the SVK repositories are created and managed over time.