Configuration Management | Data Management & Warehousing

Our Toolbox – Essential Software

Posted on 21 July 2012 by David M. Walker Posted in Techniques 3 Comments

A data warehouse project will usually involve many pieces of software.

These can be broken into two categories – architectural software and supporting tools. The architectural software will include the database, ETL tool, reporting tools, etc. but in this post I am going to look at the supporting tools we use.

I should say from the outset that by choice we have a shared Linux server and that the team like to use Apple Macs rather than Microsoft Windows machines and so our tools reflect the platforms we use. We often work inside organisations that are completely Microsoft Windows-based and so we are careful not to work in any way that would preclude interoperability with our clients environments.

Server Software

On the server we need to have the following features

Wiki
Version Control System
Ticketing System

To support this we use Subversion (also known as SVN) for version control and Trac which provides the ticketing and wiki functionality, both are open source projects and so are free. We configure them using our Project Services guidelines. Trac requires a server side database for which we normally use MySQL and phpMyAdmin to manage the MySQL database.

If we are going to be doing any R statistical analysis then we also put the server edition of RStudio following these installation instructions.

Hosted Tools

There are some hosted services that are quite useful

SurveyMonkey allows quick and easy surveys, often used for prioritising deliveries, etc.

Desktop Software

On our desktops we have a range of software

Black Pixels Versions (US$59) – a graphical user interface for SVN (OS X comes with a free command line version but we like the easy graphical interface)
CA Technologies ErWin, Embarcadero ER/Studio or Sybase PowerDesigner – Data modelling tools usually defined by client standards
Emtec ZOC (£55.12) – an easy to use SSH/Telnet Client and Terminal Emulator that we can use to connect to most servers
MacroMates TextMate (€45.63) – an amazing text editor that is really easy to use, features multiple programming language support, etc.
Microsoft Office (Word, Excel, PowerPoint, Project, Visio) – simply because you need it to work with client documentation
mSevenSoftware mSecure (Desktop £14.01, iPhone £6.99) – Password Manager – essential for maintaining a list of all the passwords etc.
Parallels (£64.99) or VMWare Fusion (£39.99) – Windows Virtual Machine manager enables us to run any Windows software if required
RStudio (Free) – a graphical interface for the R statistics package
Skype (Free) – Chat, Voice and Video conferencing
SQuirreL SQL Client (Free) – a graphical SQL client that can connect to most databases

Emerging Technologies

Tadpole (blog) – a new SQL interface similar to SQuirreL SQL, in chinese, but it useful for creating an Entity Relationship Diagram from an existing database

Version Control on Rails

Posted on 11 October 2009 by Posted in Editorial Leave a comment

In my consultancy assignments, I usually work in large organisations that have outsourced IT departments and very formal processes for everything. One aspect of this is version control.

In such an organisation every document contains a long table near the beginning with a row for every change, who made the change and when it was made and of course there will be a version number for every change. Usually the version number has at least 2 parts.

Also, the file name will have the version number in it, such as: “Some Formal Document v1.3a.doc”.

Traditionally (over the last 15 years or so) the files are stored in a Windows share. There will also be copies on local drives on various individuals’ PCs. How often do we find different people have different versions and they all believe them to be the latest?

Now, many organisations are waking up to the benefits of version control systems. However, they usually do two stupid things. The first is that they buy a version control system and the second is that they use their expensive version control system as if it were a Windows share.

You don’t need to buy a version control system. You almost certainly don’t need to buy a “Software Configuration Management” system, nor do you need to buy an “Application Lifecycle Management” system. The best version control systems are free and they are called Subversion and CVS. Using these tools is easy and simple. Discipline is necessary to get the best out of them, but you don’t need a massive amount of training. If you are really keen, you could read Pragmatic Version Control using Subversion. (You are really keen, aren’t you?)

While you’re reading the book you could minimise your chances of going off the rails by following the simple recommendations listed below:

Let the version control system do the work!
Remember that the version in the version control system is the master version; any file on your local disk is not the master until you check it in to the version control system.
Do not change the name of a file from one version to another; all versions of one file go in the same folder, under the same name (then you can see the change history).
If you do need to change the name of a file or folder, do it in the the version control system client, not on the working file on your local disk. (If you do it in the version control system it will change the working file for you and everything will stay in sync.)
Never, never, never put a version number, or date, or time in a filename – this is a version control system, not a Windows share!
Check in atomic units – one component per file – do not check in zip files or tar files
The version control system version information is trusted over any version information recorded inside a file – so do not record it inside the file.

So what has this got to do with Rails? Well, there is the DRY principle, to start with:

Every piece of knowledge in the system should be expressed in just one place.

The version control system keeps a log of your versions, so don’t keep an incomplete and out-of-date one in the document – delete that big table from the front of your document – you really do not need it. And do not put the version number in the document or in the file name. This is just repeating yourself and if you do that you will end up with many conflicting versions of the truth.

Every time someone works with the wrong version of a document, they waste company resources. Don’t tolerate it.

This article was originally published on BIonRails, another Data Management & Warehousing website

Ready for Change?

Posted on 3 June 2008 by David M. Walker Posted in Presentations Leave a comment

A presentation given to the 3rd Citia BTC Conference in Kiev.

This presentation looks at the problems of creating a long-term sustainable data warehousing environment in the context of the change that will affect them:

Data warehouses are constantly in a state of flux

The cost of build is small by comparison with the operational cost needed to support and change the system. This requires full-time long-term resourcing

There are seven types of change affecting the data warehouse :

User Driven Change (Enhancements & Requirements)
External Systems Change
Internal Systems Change
Risk Management
Data Stewardship
Alert & Trend Monitoring
Issue Management

Download Ready For Change? now

Project Services HowTo

Posted on 8 April 2007 by David M. Walker Posted in Techniques Leave a comment

Objective

Project Services is an integration of software and doman knowledge. Data Management & Warehousing has an approach to building data warehouses. This approach includes a number of templates and white papers including approaches to architecture, governance and documentation.

In order to run a project using this approach it is useful to have a change control system, a task and issue tracking system, a wiki, etc. This document describes how to build the software aspects of the solution

Installed Software Solution

The objective is to build a system using the following packages:

Server

Subversion
http://subversion.tigris.org/
Trac
http://trac.edgewall.org/
ViewVC
http://www.viewvc.org/
User_Manage
http://stein.cshl.org/~lstein/user_manage/

Client

Tortoise SVN
http://tortoisesvn.tigris.org/

Installed Software

The Data Management & Warehousing system used the following software

CentOS 4.5 fully up-to-date
Additional RPMs from http://dag.wieers.com/rpm/
- subversion-perl-1.4.4-0.1.el4.rf
- subversion-1.4.4-0.1.el4.rf
- subversion-devel-1.4.4-0.1.el4.rf
- python-clearsilver-0.10.4-2.el4.rf
- clearsilver-devel-0.10.4-2.el4.rf
- clearsilver-0.10.4-2.el4.rf
- perl-ClearSilver-0.10.4-2.el4.rf
- python-clearsilver-0.10.4-2.el4.rf
Source code downloads
- trac-0.10.4.tar.gz
- viewvc-1.0.4.tar.gz
- user_manage.tar.gz

Naming conventions used

Wherever there is a name or path that needs to be set it is written in uppercase and should be substitued appropriately

CMS_PAGE: A location of a CMS managed page e.g. http://www.datamgmt.com/?PAGE=&47
DISK_ROOT: A location not visible to the webserver e.g. /hosting/datamgmt.com/projects
DOMAIN: The domain e.g. datamgmt.com
IPADDRESS: The IP Address e.g. 10.10.10.10
PORT: The IP port e.g. 80
SUBDOMAIN: The subdomain where the projects are hosted e.g. projects.datamgmt.com
WEB_ROOT: A location visible to the webserver e.g. /hosting/datamgmt.com/website/projects

Text prefixed with a hash ‘#’ it is a unix command to be typed.
Text in italics should be edited into the appropriate file

To set up an additional project the steps in red text need to be repeated for the new project.

Mail Alias

Create a e-mail alias for the project administrators that can be used in the configuration and directed as appropriate e.g.

projects-admin@DOMAIN

Create the Subversion Repository

# mkdir -p DISK_ROOT/PROJECT/svn/

# svnadmin create --fs-type fsfs DISK_ROOT/PROJECT/svn/

Install ViewVC

# mkdir DISK_ROOT/cgi-bin  
# cp /usr/local/viewvc-1.0.4/bin/cgi/viewvc.cgi DISK_ROOT/cgi-bin

Then update the /usr/local/viewvc-1.0.4/viewvc.conf file with the following lines:

#cvs_roots 
svn_roots = PROJECT: DISK_ROOT/PROJECT/svn
default_root = svn
address = Projects Administration

Install user_manage

# cp user_manage WEB_ROOT/cgi-bin

Edit the user_manage script as follows:

$CONFIG_FILE = 'DISK_ROOT/etc/realms.conf';

# mkdir DISK_ROOT/etc

Create and edit the file DISK_ROOT/etc/realms.conf:

<Realm projects.datamgmt.com/PROJECT>
        Users      DISK_ROOT/PROJECT/etc/passwd.http
        Groups     DISK_ROOT/PROJECT/etc/group.http
        Type       Text
</Realm>

# cd DISK_ROOT/PROJECT/etc

# htpasswd -cb passwd.http USERNAME PASSWORD

Create and edit the file DISK_ROOT/PROJECT/etc/group.http

administrators: USERNAME
users: USERNAME

Configure Trac

# mkdir DISK_ROOT/tmp
# chown apache.apache DISK_ROOT/tmp
# chmod 775 DISK_ROOT/tmp

# trac-admin DISK_ROOT/PROJECT/trac initenv

Accept default values except

Project Name [My Project]> PROJECT

Path to repository [/path/to/repos]> DISK_ROOT/PROJECT/svn

Edit the configuration file DISK_ROOT/PROJECT/trac/conf/trac.ini

[header_logo]

 link = http://SUBDOMAIN

 src = /img/yourlogo.png

[notification]

 always_notify_owner = true

 always_notify_reporter = true

 smtp_default_domain = DOMAIN

 smtp_from = projects-admin@DOMAIN

 smtp_replyto = projects-admin@DOMAIN

 smtp_enabled = true

[project]

 descr = PROJECT

 name = PROJECT

 url = http://SUBDOMAIN/PROJECT/trac

[trac]

 base_url = http://SUBDOMAIN/PROJECT/trac

Configure .htaccess files

Add the following .htaccess files

Redirect the DOMAIN homepage to the main website Content Management System page for requests (This rule re-directs requests / and index.html requests)

Create and edit WEB_ROOT/.htaccess

RewriteEngine on RewriteRule ^$ CMS_PAGE RewriteRule ^index.html$ CMS_PAGE

Redirect the DOMAIN/PROJECT homepage to the Trac homepage (This rule re-directs requests / and and index.html requests)

Create and edit WEB_ROOT/PROJECT/.htaccess

RewriteEngine on RewriteRule ^$ http://DOMAIN/PROJECT/trac RewriteRule ^index.html$ http://DOMAIN/PROJECT/trac

Set up Trac security

# trac-admin DISK_ROOT/PROJECT/trac permission list

# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME user

# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME admin

# trac-admin DISK_ROOT/PROJECT/trac permission add user TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY

# trac-admin DISK_ROOT/PROJECT/trac permission add admin CONFIG_VIEW MILESTONE_ADMIN REPORT_ADMIN ROADMAP_ADMIN TICKET_ADMIN TRAC_ADMIN WIKI_ADMIN

# trac-admin DISK_ROOT/PROJECT/trac permission remove anonymous TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY

Modify other permissions to suit individual requirements

Modifications to the httpd.conf file

<VirtualHost  IPADDRESS:PORT>

ServerAdmin   projects-admin@DOMAIN
DocumentRoot  WEB_ROOT
ServerName    SUBDOMAIN
ErrorLog      WEB_ROOT/logs/errors.log
CustomLog     WEB_ROOT/logs/access.log combined env=!DontLog
ScriptAlias   /cgi-bin  WEB_ROOT/cgi-bin

   AllowOverride All

   Options ExecCGI FollowSymLinks

<Directory WEB_ROOT/PROJECT>

 AuthType Basic

 AuthName "SUBDOMAIN/PROJECT"

 AuthUserFile DISK_ROOT/PROJECT/etc/passwd.http

 AuthGroupFile DISK_ROOT/PROJECT/etc/group.http

 Require valid-user

 AllowOverride All

</Directory>

<Location /PROJECT/trac>

 SetHandler mod_python

 PythonInterpreter main_interpreter

 PythonHandler trac.web.modpython_frontend

 PythonOption TracEnv DISK_ROOT/PROJECT/trac

 PythonOption TracUriRoot /PROJECT/trac

 SetEnv PYTHON_EGG_CACHE DISK_ROOT/tmp

</Location>

<Location /PROJECT/repository>

 DAV svn

 SVNParentPath DISK_ROOT

</Location>

</VirtualHost>

Completing the installation and testing

After all of the above is done the web server needs to be restarted

# apachectl stop
# apachectl start

http://SUBDOMAIN
This should take you to the CMS page that acts as an introduction to all the projects

http://SUBDOMAIN/cgi-bin/user_manage?realm=SUBDOMAIN/PROJECT
This should take you to the user management page of the sub-domain and project.

http://SUBDOMAIN/PROJECT
This should take you to the wiki page that acts as the front page of the individual project

http://SUBDOMAIN/PROJECT/cgi-bin/viewvc.cgi
This should take you to the ViewVC page the views the source code repository

Other useful utilities

There are a number of useful plugins:

http://trac-hacks.org/wiki

Tip: Create a directory (e.g. /tmp/trac_plugins) and carry out the work from there.

Tip: Install the plugins globally using the –prefix=/usr (see below)

Tip: If you are installing a second project copy the trac.ini over after doing all the installs and then simply search and replace the project name instead of doing all the edits.

Tip: Once the ini admin module is installed much of the enabling etc. can be done from here.

Tip: Sometimes after installing a new plugin it does not appear. Try restarting the browser and webserver as stuff might be cached.

Tip: Sometimes the following line
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
can be added to the file /usr/lib/python2.3/site-packages/easy-install.pth causing a Server 500 error. This should be commented out

EZ Setup Utility

# wget http://peak.telecommunity.com/dist/ez_setup.py 
# python ez_setup.py --prefix=/usr

General Plugin Installation

# svn co http://PLUGIN_SVN_URL
# cd PLUGIN/(0.10)
# python setup.py egg_info
# python setup.py bdist_egg
# cd dist
# easy_install –prefix=/usr *.egg

Edit DISK_ROOT/PROJECT/conf/trac.ini

[components]
PLUGIN.* = enabled

Implemented Plugins

Account Manager

ini Admin

Master Tickets
Also note that the following patch must be applied before building

Nav Add

Rev Tree
Tried to install but have had issues so abandoned

Other useful plugins not installed

Timing and Estimation

Work Log

Issues

Sometimes after installing a plugin the server issues an error

Editing:

/usr/lib/python2.3/site-packages/easy-install.pth

and commenting out the line:

#import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Appears to fix this

Project Services

Posted on 8 March 2007 by David M. Walker Posted in Techniques Leave a comment

Welcome to Project Services

In the Data Management & Warehousing White Paper Data Warehouse Governance the importance of change control and issue tracking to a successful data warehousing project are highlighted. This paper also highlights the importance of a lightweight or agile development methodology and how using a wiki can help improve the develpment process.

Project Services offers the following:

A demonstration of a complete, consistent and co-herent set of tools for project planning, task, risk, issue and enhancement tracking, document and source code control, test management and project wiki integrated together to support a data warehouse development
A free hosted environment for consultancy clients of Data Management & Warehousing on which to run their project
A low cost hosted environment for projects not using Data Management & Warehousing consultancy services but wishing to use our approach to developing successful data warehouses.
A guide for those wishing to implement their own data warehouse project governance.

What is the Project Services platform ?

Each project gets it’s own private area in which to operate. No data is shared between projects. For example the ‘demo’ project can be found at http://projects.datamgmt.com/demo

Each project has the following features;

Subversion:
An open-source revision control system
Trac:
A web-based software project management and bug/issue tracking system that also provides an interface to Subversion and an integrated wiki
User Management:
A web-based interface to allow management of usernames, groups and passwords.
Tortoise SVN:
A client for Subversion that Subversion commands directly in Windows explorer.

Note: Since this service works over http or https it will work from within an corporate firewall

Can I see the demo ?

The demostration system is available at http://projects.datamgmt.com/demo with a username of ‘guest’ and a password of ‘guest’. At the moment there is not much available becuase it will take some time to build up a set of worthwhile examples, however we are actively working on it so please check back regularly.

If you are interested in a hosted environment the please Contact Us

Why the change of platform ?

For many years Data Management & Warehousing has recommended a cheap open source solution to this problem by implementing Bugzilla and CVS.

Whilst Bugzilla is web based, CVS normally required an SSH connection and the solution also required the use and integration of a seperate Wiki solution. Using Subversion and Trac has allowed Data Management & Warehousing to implement a fully http based solution which integrated change control, task and issue management and a project wiki in a single solution, but with all the heritage of the previous tools.

How can we put Project Services on our own server ?

All the code used to create Project Services is Open Source. Therefore anyone can build it on their own server. For our own internal documentation and in case anyone else is interested we have created the Project Services HowTo

Installing CVS on Windows

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

Installing CVS on Windows

To access and work with the the CVS repositories you must install a CVS client on your Windows PC that can talk to the CVS servers. There are scores of CVS clients available, but at a minimum you must install the CVS command line client.

This document also includes the instructions needed to install two GUI Windows clients for CVS: WinCvs and TortoiseCVS.

Use the instructions in this document to install the CVS command-line client and (optionally) one or both of the GUI clients on a Windows PC.

Install CVS Command-Line Client

First use the Installing CVS on Windows procedure to get SSH set up and to obtain user account(s) on one or more CVS servers.
Create a directory called ‘cvs’ on a local harddrive. This directory will be referred to as ‘d:\cvs’ in the rest of these instructions.Note – When working with Windows/Unix tools, it is best to avoid directories and file names with embedded spaces. So don’t create the ‘cvs’ directory under your ‘Program Files’ folder.
Copy the cvs code to your new d:\cvs directory.
Now add the ‘d:\cvs’ directory to your SYSTEM PATH environment variable using the System option in the Windows Control Panel. Press the Apply button to make the new PATH setting take affect.
Open a DOS command window and type the ‘set’ command. Make sure (a) that your PATH now includes ‘d:\cvs’; and (b) that the CVS_RSH variable is defined and points to the location of the ‘plink’ program.
In your DOS command window, move to the directory where you want to put your CVS project directory. For example:cd /D d:\projects
Now test your CVS setup by checking out the CVS project that you will be working on using the command:cvs -d :ext:@:/home/cvs checkoutfor example, if user ‘username’ wants to checkout project ‘project’ from the CVS server ServerName the correct command is:cvs -d :ext:username@ServerName:/home/cvs checkout project
You should get a ‘cvs server: Updating ‘ message and CVS will put a working copy of the entire project in your current directory.
You can then use any tools you want to make and test project changes. Use the ‘cvs add’ command to notify CVS of any new directories and files you add to the project. Use ‘cvs commit’ to submit your changes back into CVS once you have completed coding and testing.
If you are new to CVS, a good tutorial can be found in chapter two of Karl Fogel’s book Open Source Development with CVS. Of particular use for beginners are the sections CVS Basics and A Day With CVS. A copy of this book and of the CVS manual itself (called the Cederquist), were put in your d:\cvs directory when you installed the CVS command-line client above.

Installing the WinCvs Client

In order to work properly, WinCvs requires that you install both Python and Tcl on your PC first. You will also need a good text editor and a good diff program. Instructions for Python, Tcl and ExamDiff program installation are included in this section.
Install Python:
- Download and run the latest Python EXE in this directory. This will launch a standard Windows installer. Follow the instructions to install Python on your PC.
- When installing Python, the only component required for WinCvs is the Python interpreter and libraries. You can exclude all the remaining components from the install.
Install Tcl:
- Download and run the latest ActiveTcl EXE in this directory. This will launch a standard Windows installed. Follow the install prompts to complete the installation.
Install ExamDiff:
- Create a directory for ExamDiff on your PC. For example: d:\examdiff or d:\Program Files\examdiff.
- Download ExamDiff to your local ExamDiff directory.
Now begin the WinCvs install. Download and run the SETUP.EXE program to install WinCvs on your PC.

Configuring WinCvs to Connect to CVS Server

After installing WinCvs, run it from your Start–>Program–>GNU–>WinCvs menu. WinCvs will automatically open the ‘WinCvs Preferences’ dialog the first time it is run.
Under the ‘GENERAL’ tab, set ‘Authentication’ to ‘SSH’. Click on the SETTINGS button and enter additional SSH information:
- Check the ‘If SSH is not in the PATH’ box and enter the full path of the plink program – ‘d:\putty\plink’.
- Check the ‘Additional SSH options’ box and enter ‘-ssh’.
- Click on OK to save your SSH settings.
- In the ‘Path’ field enter: ‘/home/cvs’ (this is the location of the CVS repository on the CVS server).
- In the ‘Host Address’ field enter the name of your CVS server.
- In ‘User Name’ enter your network logon (lowercase). For example, ‘username’.
Under the ‘GLOBALS’ tab, uncheck the ‘Checkout read-only’ option.
Under the ‘WINCVS’ tab, set the following options:
- In the ‘HOME…’ field enter a home directory name such as ‘d:\home\WinCvs’. (WinCvs will create the directory if it does not already exist.)
- Check the ‘External diff program’ box and enter the location of the ExamDiff executible you installed earlier.
- Under ‘Default viewer used to open files’ enter the full path of your favorite text editor, such as Textpad.
Click on the OK button to exit the WinCvs Preferences dialog.

Getting Started with WinCvs – Documentation

There are two documents that can help you get started with WinCvs:

WinCVS Daily Use Guide a quick overview of using WinCvs
WinCvs-1.3.pdf the WinCvs reference manual

The “WinCvs Daily Use Guide” is probably the best place to start.

Installing the TortoiseCVS GUI

TortoiseCVS is a plug-in for the Windows Explorer. It gives you easy, direct access to CVS commands from the Explorer. Even with WinCvs installed you will still probably find TortoiseCVS handy to have installed on your PC.

Installing TortoiseCVS:

Install TortoiseCVS on your PC by downloading and running the latest version of the TortoiseCVS.
Because TortoiseCVS is an Explorer plug-in, you will need to REBOOT your PC after running the Tortoise installer.
Next go to Start–>Programs–>TortoiseCVS–>Preferences to configure a connection to the CVS server.
- Under the MAIN tab, select the ExamDiff program as your ‘External diff application’.
- Under the SSH tab, enter the name of your ssh application – normally this will be ‘d:\putty\plink.exe’.
- Click on OK to save your preferences.
Now open Windows Explorer and go the the directory where you plan to put your CVS projects. Right-click in that directory and select the ‘CVS Checkout’ option.
Under the MODULE tab make the following changes:
- Make sure ‘Protocol’ is set to ‘Internet (secure shell)’.
- Set ‘Server’ to ‘ServerName’ as appropriate.
- Set ‘Repository directory’ to ‘/home/cvs’.
- Set ‘User name’ to your network logon id (lowercase!).
- Set ‘Module’ to the name of the CVS project you want to work on. (You can click on the FETCH LIST button and TortoiseCVS will populate the drop-down box with a list of all valid CVS projects on the server.)
- Click on OK to proceed with the checkout.
TortoiseCVS will pull a copy of the requested CVS project down to your PC and you can start working on it.
For more information how to use TortoiseCVS, please read the TortoiseCVS Support Documentation

Using WinCvs & TortoiseCvs Together

You can use WinCvs and TortoiseCvs (and the CVS command-line) interchangeably. For example, you can checkout a project using TortoiseCvs, work on it with WinCvs and then commit it using the CVS command-line.

As a rule this should also hold true for any other CVS clients you might decide to install on your PC. Possible exception: clients built into IDEs like Forte and Eclipse.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

Configuring SSH Keys

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

Overview

Once you have installed PuTTY but before you can connect to the CVS server, you must create your SSH authentication keys.
SSH authentication keys are a more secure way of controlling access to the CVS server. Once configured they will allow you to connect to the CVS server without entering a log-on name and password – this is important since most of the time your connections to the CVS server will actually be made by your CVS client and these CVS-to-CVS connections could be disrupted by prompts for user name and
password.

SSH key authentication requires that you create two keys. One is your
PUBLIC key. This is a key you can share with anyone. This key will be stored on the CVS server. The second key is your PRIVATE key. You must make sure you keep your private key secure. This is especially true if you are using a laptop, working off-site, or working from home.

Instructions

First open a DOS shell (CMD.EXE) and create a .ssh directory in your putty directory
```
      cd /D d:\putty
      mkdir .ssh
```
Note – Windows Explorer will not allow you to create a directory that starts with a period.

Run the PuTTYGen program to create your SSH authentication keys.

For Type of key to generate select SSH2 RSA.

The Number of bits in a generated key should be set to 1024.

Click on the GENERATE button to start key generation.

A progress bar will appear and you will be asked to move your mouse around to ‘generate randomness’. Move your mouse around until the first progress bar fills up. Then key generation will start and may take a minute or two.

A new dialog box will appear with your public key when the key generation completes.

You should pre-pend ServerName to the Key Comment box. This will make life easier if you add additional SSH keys in the future for other purposes.

Next type in a Key passphrase. This should be a phrase of three or more ‘words’ that will be used to encrypt and protect your private key. Don’t use dictionary words and try to mix numbers and upper/lower-case letters in your pass phrase.One recommended technique is to think of a phase you will find easy to remember, then deliberately introduce some misspellings and numbers into the words.Warning – do not forget your key pass phrase. There is no way to recover it. If you forget it, you will have to generate a new pair of SSH keys.

Now click on the SAVE PUBLIC KEY button and save your public key to the file: d:\putty\.ssh\identity.pub.

Next click on the SAVE PRIVATE KEY button and save your private key to the file: d:\putty\.ssh\identity.

Use your mouse to select your entire public key from the Public key for pasting into authorized_keys file box. The public key information will be automatically copied to the Windows clipboard.

Now install your new SSH public key into the appropriate area on the Linux Server.

Use the CLOSE button at the top-right of the PuTTY Key Generator window to exit from the PuTTYGen program.

Return to the Installing the PuTTY SSH Client for Windows instructions to complete configuration of SSH.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

Installing the PuTTY SSH Client for Windows

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

Overview

PuTTY is a GUI Windows client that makes it easy to use the SSH (secure shell) protocol to connect your Windows PC to a Linux server hosting a CVS repository

Installing an SSH client and generating your personal SSH keys must be done before you can be given a user account to connect to the CVS server.

Why PuTTY?

There are a number of free and commercial SSH clients available for Windows. These instructions tell you how to install and configure one of these clients: PuTTY.
PuTTY is commonly selected because it is a self-contained product with native Win32 binaries that includes an easy-to-use GUI interface and offers very robust support for the SSH2 protocol.
If you already use the Cygwin (Unix-for-windows) product, you may wish to use the OpenSSH client that comes with Cygwin instead PuTTY. However, detailed instructions for this option are not included in this document.

Installing PuTTY

Create a directory called putty on your C or D drive. This will be referred to as d:\putty in the instructions that follow.

Download the code from http://www.chiark.greenend.org.uk/~sgtatham/putty/ to your new d:\putty folder.

Now run the putty.exe program. The PuTTY Configuration dialog will be displayed.

Under the Session category, set these options:

          Host Name --> Server you are connecting to
          Protocol  --> SSH

Select the Terminal option in the left pane and turn on the option Use background colour to erase screen.

Select the Connection option in the left pane and enter your user name in the Auto-login username box.Remember, Linux is case sensitive, be sure you enter your user name in lowercase since this is the way the CVS Administrator will normally set up your account on the CVS server.

Next select the SSH option under the Connection category in the left pane. Under Preferred SSH protocol version select 2.

Select the Auth option and de-select the Attempt keyboard-interactive authentication (SSH2) option.

Now go back the Session category and type ServerName into the Saved Sessions box as appropriate. Click on the SAVE button to store your CVS session options.

Configure Connection to CVS Server

Next use the instructions in Configure SSH Keys to create and install your SSH authentication keys.

Install your SSH public key on the CVS server. Now you can configure PuTTY to use your SSH keys to make the connection to the CVS server.

Run PuTTY.exe and click on the LOAD button to load your ServerName settings as appropriate.

Choose the Connection–>SSH–>Auth option in the left pane and enter the name of your private key file into the box shown in the right pane. The name of this file will be something like:
```
          d:\putty\.ssh\identity.
```
Click on the Session category in the left pane and use the SAVE button to save your updated ServerName session settings.
Now click on OPEN to try establishing a connection to the CVS server. The first time you connect, you should get a PuTTY security alert message that gives you the fingerprint of the CVS server.
If you do see the correct fingerprint, click on YES. PuTTY will store this fingerprint in the Windows registry and you won’t get this prompt on future connections.

PuTTY will now connect to the CVS server. You should see messages like these:

          Using username your-user-name.
          Authenticating with public key "ServerName Server rsa-key-yyyymmdd"
          Pass-phrase for key "... rsa-key-yyyymmdd:

Enter your pass-phrase and you will be logged in to the CVS server. If you cannot log-on to the CVS server, contact the CVS Administrator and ask the admin to check your user account on the CVS server.

Configure SSH Agent for CVS Client/Server Connections

Now you can configure PuTTY to automatically connect you to the CVS server using your authentication keys instead of prompting you for your SSH pass-phrase each time. By using the SSH agent program called pageant, you will only have to enter your SSH pass-phrase once when you boot your PC. The SSH Agent will then keep your SSH key loaded in memory and automatically provide it each time your CVS client(s) need to connect to the CVS server until the next time you reboot.

Using Windows Explorer, go to your d:\putty directory. Right-click on the pageant.exe program and create a shortcut.

Right-click on the your new pageant shortcut.
1. Use the RENAME option to change the name of the
  shortcut to SSH-Agent.
2. Use the PROPERTIES option and click on the SHORTCUT tab. Add the name of your private key file to the pageant run command in the TARGET box so that the run command looks like this:
```
              d:\putty\pageant.exe d:\putty\.ssh\identity
```
3. Click on the OK button to save your changes.
Now double-click on your SSH-Agent shortcut to run the pageant program.
The pageant program will prompt you for the pass-phrase for your SSH key. Enter your pass phrase and click on OK.
You should now see the pageant icon in your system tray at the bottom right-hand corner of your screen.
Right-click on the pageant icon and select the VIEW KEYS option. Pageant will open a window with a list of the SSH keys it has loaded. You should see the fingerprint for your private key listed in this window.
Close the pageant key list window.
Now double-click on the putty.exe program again and ask it to connect to the CVS server. This time putty should automatically connect to the CVS server. You should see the following messages:
```
          Using username your-user-name.
          Authenticating with public key "CVS Server rsa-key-yyyymmdd" from agent
          Last login: date-and-time from your-workstation-name
```
By the way, the last login message is also a security aid. If you notice a last login time or workstation that are not yours, please notify the CVS Administrator immediately.

You may want to copy your SSH-Agent shortcut to your Windows startup folder so that it will automatically run and load your SSH key(s) each time you reboot your PC.

Configure Plink for Background Connections

The final step in PuTTY configuration is to set up the plink tool so that it can be used for automatic background connections from your PC to the CVS server.

First make two changes to the environment variables on your
PC:
1. Add the d:\putty directory to your system PATH environment variable using the System option in the Windows Control Panel.

Create a new system environment variable called CVS_RSH with a value of d:\putty\plink.
Reboot your PC to make the new environment settings take effect.

Now test plink. Open a DOS command shell and enter the command:
```
          plink ServerName echo $HOSTNAME $USER $HOME
```
You should see a response that looks something like this:
```
          ServerName username /home/username
```
Note – The ServerName in the plink command is the name of the PuTTY session configuration you saved earlier in this procedure.
Note – If you have trouble connecting with plink try this command instead:
```
           plink -v ServerName echo $HOSTNAME $USER $HOME
```
The -v option will cause plink to write diagnostic messages as it connects to the CVS server. These messages can help you pin down the problem with your connection.

Congratulations, you have now completed all the SSH setup
needed connect to the CVS server.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

CVS – A Basic Introduction

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

CVS – Getting Started

Use this document to get started using CVS. It contains a very short intro to CVS and points you to the more detailed installation and configuration procedures you must follow to use CVS from a Windows or Linux PC.

These instructions are written for developers who need CVS access for the first time.

A Brief Intro to CVS

CVS is short for Concurrent Versions System.

CVS is a widely used open source source-code control system. It operates using a client-server model. The source-code is stored in a CVS repository on the CVS server. You interact with the CVS repository by using a CVS client on your PC. You use CVS commands like checkout, update, and commit make your source-code changes locally and then communicate them back to the central CVS repository.

From a development point of view CVS is different from many other version-control tools in two key aspects.

First, when using CVS you normally work at a project level. Rather than checking out a single file, you checkout an entire project directory tree. For example, if you have a project that looks like this:

  \mystuff
    dirA\
      fileA1
      fileA2
    dirB\
      fileB1
      fileB2

And you know you need to change fileB1, rather than just checking out that single file, you checkout out the entire mystuff project. When you do this, CVS will create the mystuff project directory with all of its sub-directories and files on your PC. You can then make your changes, compile them, and test them. When you are done, you use one CVS commit command to submit your project changes back into CVS. CVS will detect every update you have made within the project, and will submit new versions of those files back into the repository.

Second, implicit in our discussion above is another key difference when using CVS. When you checkout source from CVS, it is not locked. CVS allows concurrent updates to be made to project code. At the same time that you are modifying fileB1 another developer can also be changing the same file. When you commit your changes back to the CVS repository, CVS will automatically merge all non-conflicting changes. If it finds any conflicting changes, CVS will notify you of the conflict and will not submit that file back to the repository until you have manually fixed the conflicting change(s).

One final important point to make about CVS – you never work directly within the CVS repository. Instead you check out projects to your own PC. By doing this you create private work areas (or ‘sandboxes’). Once you’ve got your sandbox, you can edit, compile and test completely isolated from changes being made to the CVS repository by other users. You can bring your sandbox up-to-date with the central repository at any time using the CVS update command. The timing and frequency of these updates is completely up to you and your project lead. Each time you complete a round of changes and reach a ‘turnover’ point, commit the entire project back into CVS and then begin work on the next round of changes.

To Learn More About CVS

If you are new to CVS, there is an almost overwhelming amount of information on the web. But the best place to start is with Karl Fogel’s book Open Source Development with CVS. Read the chapter called An Overview of CVS, especially the sections Basic Concepts and A Day With CVS.

The CVS manual, Version Management with CVS, also known as the Cederqvist is also available on-line. However, it is written much more as a reference manual than a tutorial.

Getting Started – Step by Step

Your first step is to determine which of the CVS server you will be using, if you will be working on multiple projects, you may need access to more than one.If you will be working on an existing project, ask the project lead which CVS server is used to host that project.
Install and configure an SSH client for secure access to the CVS repository.If you are using Windows, use these instructions:
1. Installing the PuTTY SSH Client for Windows
2. Configure SSH Keys
If you are using Windows, use the Install CVS for Windows instructions to install the CVS command-line and gui clients on your PC. If you are using Linux, the CVS command-line client should have been installed by default.
You are now ready to begin working with CVS.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

Configuration Management & Issue Tracking

Posted on 2 June 2004 by David M. Walker Posted in Techniques Leave a comment

One commonly overlooked side of building a data warehouse is the need for configuration management and issue tracking. This article looks at a way to do this.

Since this article was written we have upgraded from Bugzilla to Trac and CVS to SVN and integrated in in our Project Services

Introduction

ETL tool transformations, data models, query tools & report writers, analysis documents and data quality reports are all part of the collection of information gathered during the build of a data warehouse. Once in production the tracking of issues that arise from day to day operations and the relationship to the ETL process also need tracking.

As a consultancy we normally steer clear of recommending specific products as generic solutions and only do so for particular clients who need tools to meet specific situations. However we have generally noted most organisations do not do the configuration management and issue tracking well and that commercial solutions, unless part of an existing corporate infrastructure, are costly to deploy.

If this is normally so expensive what should we do ?

The premise of this article is using open source software. Don’t stop reading – like the software this advice is free so it is worth a look.

Whatever is wrong with individual pieces of open source software the one thing that has worked well is the issue tracking and source code control. Sites such as http://sourceforge.net have over 82,000 projects and 858,000 users (as at 1st June 2004).

There are two pieces of software that stand out and are widely used for configuration management and issue tracking. These are:

CVS: CVS is a version control system and important component of Source Configuration Management (SCM) that records the history of sources files and documents.
Bugzilla: Bugzilla is one example of a class of programs called “Defect Tracking Systems”, or, more commonly, “Bug-Tracking Systems” that groups of developers to keep track of outstanding bugs in their product effectively.

What does CVS offer me ?

CVS provides a repository on a server (Windows or Linux – see later) that can store and version files (including binary files and special format files such as Microsoft Word). When you create a file you ‘check it in’ and when you want to read it again or deploy it you ‘check it out’. If you want to edit the file then you check it out in a locked mode to prevent others editing it whist you are. It is also possible mutliple developers to check out the code, work on it and check it back in and ask for conflict resolution etc. Finally you can tag ‘releases’ and have code branhes off the main development that allow new concepts to be tried without losing any of the code already developed

By default it comes with a faily standard Unix style command line interface but there are many useful client interfaces such as WinCVS that provides a graphical user interface for most platforms (Windows/Unix/Linux/Mac) and CVSWeb that provided a browser based interface to view the repository. The WinCVS site also has links to other clients and other sites about CVS. A fully downloadable book can also be found at http://cvsbook.red-bean.com/

What does Bugzilla offer me ?

Bugzilla provides a simple web and e-mail based issue tracking system that allows mutiple products (or projects) and their components to have issues raised and prioritised and tracked through to resolution.

If features :

integrated, product-based granular security schema
inter-bug dependencies and dependency graphing
advanced reporting capabilities
a robust, stable RDBMS back-end
extensive configurability
a very well-understood and well-thought-out natural bug resolution protocol
email, XML, console, and HTTP APIs
available integration with automated software configuration management systems, including Perforce and CVS (through the Bugzilla email interface and checkin/checkout scripts)

So where is the catch ?

There isn’t one, all you need is a small Windows or Linux server that runs a web server such as IIS or Apache with enough disk to hold your repositories. Our 10,000 issue bugzilla database takes 5Mb and we have about 1Gb or source code from mutliple projects. A machine of entry level standard with a modern but not superfast processor, 0.5Gb memory and 100Gb hard disk and a backup device (recordable DVD or tape drive) with an OS of your choice will easily support most organisations.

You may need some help configuring the tools but the download and installation are easily followed from the various websites and you can always Google CVS or Google Bugzilla

What should we put into CVS ?

The CVS repository should hold everything you need to (re-)build your data warehouse this will include:

Requirement Specifications
Analysis and Design Documents
Code
Data Models (if you use a tool such as ErWin or Oracle Designer take an export of the repository and check it in after you have made any changes
Universes and End User Layers (if you use a tool such as Business Objects or Oracle Discoverer export the metadata for the business layer and report definitions)
System Configuration scripts (how the systems and databases that hold your data warehouse are built)
Manuals and user/system docuemntation
ETL Metadata exports
Any other metadata
Any other documents

What should we put into Bugzilla ?

The Bugzilla repository should hold everything you need to track issues, changes and enhancements to your data warehouse this will include:

All issues (prioritised via the tool)
All enhancement requests
All dependencies
Links to the CVS repository
Comments and observations about the issue
Issue status
Anything else that relates to a bugzilla feature

Conclusion

If your orgainsation has well organised procedures and development practices with existing tools, has a large budget to buy tools and external consultants to deploy them, or is so completely adverse to the fact that something available for free via open source could be useful then our plea in the second paragraph to ‘not stop reading … this advice is free’ was misguided.

If, however, your project:

has no budget for this type of facility but you’ve realised you desparately need it
has lost a day/week/month’s worth of work and don’t want to make the same mistake after you have recovered
has an enthusiastic techie who is willing to set it up and have a go
needs to prove it is in control of the development cycle to the project office/auditors
just wants to do the right thing

then hopefully this article has given you some pointers, and of course if you want some further help you can also contact us