CVS | Data Management & Warehousing

Installing CVS on Windows

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

Installing CVS on Windows

To access and work with the the CVS repositories you must install a CVS client on your Windows PC that can talk to the CVS servers. There are scores of CVS clients available, but at a minimum you must install the CVS command line client.

This document also includes the instructions needed to install two GUI Windows clients for CVS: WinCvs and TortoiseCVS.

Use the instructions in this document to install the CVS command-line client and (optionally) one or both of the GUI clients on a Windows PC.

Install CVS Command-Line Client

First use the Installing CVS on Windows procedure to get SSH set up and to obtain user account(s) on one or more CVS servers.
Create a directory called ‘cvs’ on a local harddrive. This directory will be referred to as ‘d:\cvs’ in the rest of these instructions.Note – When working with Windows/Unix tools, it is best to avoid directories and file names with embedded spaces. So don’t create the ‘cvs’ directory under your ‘Program Files’ folder.
Copy the cvs code to your new d:\cvs directory.
Now add the ‘d:\cvs’ directory to your SYSTEM PATH environment variable using the System option in the Windows Control Panel. Press the Apply button to make the new PATH setting take affect.
Open a DOS command window and type the ‘set’ command. Make sure (a) that your PATH now includes ‘d:\cvs’; and (b) that the CVS_RSH variable is defined and points to the location of the ‘plink’ program.
In your DOS command window, move to the directory where you want to put your CVS project directory. For example:cd /D d:\projects
Now test your CVS setup by checking out the CVS project that you will be working on using the command:cvs -d :ext:@:/home/cvs checkoutfor example, if user ‘username’ wants to checkout project ‘project’ from the CVS server ServerName the correct command is:cvs -d :ext:username@ServerName:/home/cvs checkout project
You should get a ‘cvs server: Updating ‘ message and CVS will put a working copy of the entire project in your current directory.
You can then use any tools you want to make and test project changes. Use the ‘cvs add’ command to notify CVS of any new directories and files you add to the project. Use ‘cvs commit’ to submit your changes back into CVS once you have completed coding and testing.
If you are new to CVS, a good tutorial can be found in chapter two of Karl Fogel’s book Open Source Development with CVS. Of particular use for beginners are the sections CVS Basics and A Day With CVS. A copy of this book and of the CVS manual itself (called the Cederquist), were put in your d:\cvs directory when you installed the CVS command-line client above.

Installing the WinCvs Client

In order to work properly, WinCvs requires that you install both Python and Tcl on your PC first. You will also need a good text editor and a good diff program. Instructions for Python, Tcl and ExamDiff program installation are included in this section.
Install Python:
- Download and run the latest Python EXE in this directory. This will launch a standard Windows installer. Follow the instructions to install Python on your PC.
- When installing Python, the only component required for WinCvs is the Python interpreter and libraries. You can exclude all the remaining components from the install.
Install Tcl:
- Download and run the latest ActiveTcl EXE in this directory. This will launch a standard Windows installed. Follow the install prompts to complete the installation.
Install ExamDiff:
- Create a directory for ExamDiff on your PC. For example: d:\examdiff or d:\Program Files\examdiff.
- Download ExamDiff to your local ExamDiff directory.
Now begin the WinCvs install. Download and run the SETUP.EXE program to install WinCvs on your PC.

Configuring WinCvs to Connect to CVS Server

After installing WinCvs, run it from your Start–>Program–>GNU–>WinCvs menu. WinCvs will automatically open the ‘WinCvs Preferences’ dialog the first time it is run.
Under the ‘GENERAL’ tab, set ‘Authentication’ to ‘SSH’. Click on the SETTINGS button and enter additional SSH information:
- Check the ‘If SSH is not in the PATH’ box and enter the full path of the plink program – ‘d:\putty\plink’.
- Check the ‘Additional SSH options’ box and enter ‘-ssh’.
- Click on OK to save your SSH settings.
- In the ‘Path’ field enter: ‘/home/cvs’ (this is the location of the CVS repository on the CVS server).
- In the ‘Host Address’ field enter the name of your CVS server.
- In ‘User Name’ enter your network logon (lowercase). For example, ‘username’.
Under the ‘GLOBALS’ tab, uncheck the ‘Checkout read-only’ option.
Under the ‘WINCVS’ tab, set the following options:
- In the ‘HOME…’ field enter a home directory name such as ‘d:\home\WinCvs’. (WinCvs will create the directory if it does not already exist.)
- Check the ‘External diff program’ box and enter the location of the ExamDiff executible you installed earlier.
- Under ‘Default viewer used to open files’ enter the full path of your favorite text editor, such as Textpad.
Click on the OK button to exit the WinCvs Preferences dialog.

Getting Started with WinCvs – Documentation

There are two documents that can help you get started with WinCvs:

WinCVS Daily Use Guide a quick overview of using WinCvs
WinCvs-1.3.pdf the WinCvs reference manual

The “WinCvs Daily Use Guide” is probably the best place to start.

Installing the TortoiseCVS GUI

TortoiseCVS is a plug-in for the Windows Explorer. It gives you easy, direct access to CVS commands from the Explorer. Even with WinCvs installed you will still probably find TortoiseCVS handy to have installed on your PC.

Installing TortoiseCVS:

Install TortoiseCVS on your PC by downloading and running the latest version of the TortoiseCVS.
Because TortoiseCVS is an Explorer plug-in, you will need to REBOOT your PC after running the Tortoise installer.
Next go to Start–>Programs–>TortoiseCVS–>Preferences to configure a connection to the CVS server.
- Under the MAIN tab, select the ExamDiff program as your ‘External diff application’.
- Under the SSH tab, enter the name of your ssh application – normally this will be ‘d:\putty\plink.exe’.
- Click on OK to save your preferences.
Now open Windows Explorer and go the the directory where you plan to put your CVS projects. Right-click in that directory and select the ‘CVS Checkout’ option.
Under the MODULE tab make the following changes:
- Make sure ‘Protocol’ is set to ‘Internet (secure shell)’.
- Set ‘Server’ to ‘ServerName’ as appropriate.
- Set ‘Repository directory’ to ‘/home/cvs’.
- Set ‘User name’ to your network logon id (lowercase!).
- Set ‘Module’ to the name of the CVS project you want to work on. (You can click on the FETCH LIST button and TortoiseCVS will populate the drop-down box with a list of all valid CVS projects on the server.)
- Click on OK to proceed with the checkout.
TortoiseCVS will pull a copy of the requested CVS project down to your PC and you can start working on it.
For more information how to use TortoiseCVS, please read the TortoiseCVS Support Documentation

Using WinCvs & TortoiseCvs Together

You can use WinCvs and TortoiseCvs (and the CVS command-line) interchangeably. For example, you can checkout a project using TortoiseCvs, work on it with WinCvs and then commit it using the CVS command-line.

As a rule this should also hold true for any other CVS clients you might decide to install on your PC. Possible exception: clients built into IDEs like Forte and Eclipse.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

Configuring SSH Keys

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

Overview

Once you have installed PuTTY but before you can connect to the CVS server, you must create your SSH authentication keys.
SSH authentication keys are a more secure way of controlling access to the CVS server. Once configured they will allow you to connect to the CVS server without entering a log-on name and password – this is important since most of the time your connections to the CVS server will actually be made by your CVS client and these CVS-to-CVS connections could be disrupted by prompts for user name and
password.

SSH key authentication requires that you create two keys. One is your
PUBLIC key. This is a key you can share with anyone. This key will be stored on the CVS server. The second key is your PRIVATE key. You must make sure you keep your private key secure. This is especially true if you are using a laptop, working off-site, or working from home.

Instructions

First open a DOS shell (CMD.EXE) and create a .ssh directory in your putty directory
```
      cd /D d:\putty
      mkdir .ssh
```
Note – Windows Explorer will not allow you to create a directory that starts with a period.

Run the PuTTYGen program to create your SSH authentication keys.

For Type of key to generate select SSH2 RSA.

The Number of bits in a generated key should be set to 1024.

Click on the GENERATE button to start key generation.

A progress bar will appear and you will be asked to move your mouse around to ‘generate randomness’. Move your mouse around until the first progress bar fills up. Then key generation will start and may take a minute or two.

A new dialog box will appear with your public key when the key generation completes.

You should pre-pend ServerName to the Key Comment box. This will make life easier if you add additional SSH keys in the future for other purposes.

Next type in a Key passphrase. This should be a phrase of three or more ‘words’ that will be used to encrypt and protect your private key. Don’t use dictionary words and try to mix numbers and upper/lower-case letters in your pass phrase.One recommended technique is to think of a phase you will find easy to remember, then deliberately introduce some misspellings and numbers into the words.Warning – do not forget your key pass phrase. There is no way to recover it. If you forget it, you will have to generate a new pair of SSH keys.

Now click on the SAVE PUBLIC KEY button and save your public key to the file: d:\putty\.ssh\identity.pub.

Next click on the SAVE PRIVATE KEY button and save your private key to the file: d:\putty\.ssh\identity.

Use your mouse to select your entire public key from the Public key for pasting into authorized_keys file box. The public key information will be automatically copied to the Windows clipboard.

Now install your new SSH public key into the appropriate area on the Linux Server.

Use the CLOSE button at the top-right of the PuTTY Key Generator window to exit from the PuTTYGen program.

Return to the Installing the PuTTY SSH Client for Windows instructions to complete configuration of SSH.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

CVS – A Basic Introduction

Posted on 30 June 2004 by David M. Walker Posted in Techniques Leave a comment

CVS – Getting Started

Use this document to get started using CVS. It contains a very short intro to CVS and points you to the more detailed installation and configuration procedures you must follow to use CVS from a Windows or Linux PC.

These instructions are written for developers who need CVS access for the first time.

A Brief Intro to CVS

CVS is short for Concurrent Versions System.

CVS is a widely used open source source-code control system. It operates using a client-server model. The source-code is stored in a CVS repository on the CVS server. You interact with the CVS repository by using a CVS client on your PC. You use CVS commands like checkout, update, and commit make your source-code changes locally and then communicate them back to the central CVS repository.

From a development point of view CVS is different from many other version-control tools in two key aspects.

First, when using CVS you normally work at a project level. Rather than checking out a single file, you checkout an entire project directory tree. For example, if you have a project that looks like this:

  \mystuff
    dirA\
      fileA1
      fileA2
    dirB\
      fileB1
      fileB2

And you know you need to change fileB1, rather than just checking out that single file, you checkout out the entire mystuff project. When you do this, CVS will create the mystuff project directory with all of its sub-directories and files on your PC. You can then make your changes, compile them, and test them. When you are done, you use one CVS commit command to submit your project changes back into CVS. CVS will detect every update you have made within the project, and will submit new versions of those files back into the repository.

Second, implicit in our discussion above is another key difference when using CVS. When you checkout source from CVS, it is not locked. CVS allows concurrent updates to be made to project code. At the same time that you are modifying fileB1 another developer can also be changing the same file. When you commit your changes back to the CVS repository, CVS will automatically merge all non-conflicting changes. If it finds any conflicting changes, CVS will notify you of the conflict and will not submit that file back to the repository until you have manually fixed the conflicting change(s).

One final important point to make about CVS – you never work directly within the CVS repository. Instead you check out projects to your own PC. By doing this you create private work areas (or ‘sandboxes’). Once you’ve got your sandbox, you can edit, compile and test completely isolated from changes being made to the CVS repository by other users. You can bring your sandbox up-to-date with the central repository at any time using the CVS update command. The timing and frequency of these updates is completely up to you and your project lead. Each time you complete a round of changes and reach a ‘turnover’ point, commit the entire project back into CVS and then begin work on the next round of changes.

To Learn More About CVS

If you are new to CVS, there is an almost overwhelming amount of information on the web. But the best place to start is with Karl Fogel’s book Open Source Development with CVS. Read the chapter called An Overview of CVS, especially the sections Basic Concepts and A Day With CVS.

The CVS manual, Version Management with CVS, also known as the Cederqvist is also available on-line. However, it is written much more as a reference manual than a tutorial.

Getting Started – Step by Step

Your first step is to determine which of the CVS server you will be using, if you will be working on multiple projects, you may need access to more than one.If you will be working on an existing project, ask the project lead which CVS server is used to host that project.
Install and configure an SSH client for secure access to the CVS repository.If you are using Windows, use these instructions:
1. Installing the PuTTY SSH Client for Windows
2. Configure SSH Keys
If you are using Windows, use the Install CVS for Windows instructions to install the CVS command-line and gui clients on your PC. If you are using Linux, the CVS command-line client should have been installed by default.
You are now ready to begin working with CVS.

Credits

This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.

Configuration Management & Issue Tracking

Posted on 2 June 2004 by David M. Walker Posted in Techniques Leave a comment

One commonly overlooked side of building a data warehouse is the need for configuration management and issue tracking. This article looks at a way to do this.

Since this article was written we have upgraded from Bugzilla to Trac and CVS to SVN and integrated in in our Project Services

Introduction

ETL tool transformations, data models, query tools & report writers, analysis documents and data quality reports are all part of the collection of information gathered during the build of a data warehouse. Once in production the tracking of issues that arise from day to day operations and the relationship to the ETL process also need tracking.

As a consultancy we normally steer clear of recommending specific products as generic solutions and only do so for particular clients who need tools to meet specific situations. However we have generally noted most organisations do not do the configuration management and issue tracking well and that commercial solutions, unless part of an existing corporate infrastructure, are costly to deploy.

If this is normally so expensive what should we do ?

The premise of this article is using open source software. Don’t stop reading – like the software this advice is free so it is worth a look.

Whatever is wrong with individual pieces of open source software the one thing that has worked well is the issue tracking and source code control. Sites such as http://sourceforge.net have over 82,000 projects and 858,000 users (as at 1st June 2004).

There are two pieces of software that stand out and are widely used for configuration management and issue tracking. These are:

CVS: CVS is a version control system and important component of Source Configuration Management (SCM) that records the history of sources files and documents.
Bugzilla: Bugzilla is one example of a class of programs called “Defect Tracking Systems”, or, more commonly, “Bug-Tracking Systems” that groups of developers to keep track of outstanding bugs in their product effectively.

What does CVS offer me ?

CVS provides a repository on a server (Windows or Linux – see later) that can store and version files (including binary files and special format files such as Microsoft Word). When you create a file you ‘check it in’ and when you want to read it again or deploy it you ‘check it out’. If you want to edit the file then you check it out in a locked mode to prevent others editing it whist you are. It is also possible mutliple developers to check out the code, work on it and check it back in and ask for conflict resolution etc. Finally you can tag ‘releases’ and have code branhes off the main development that allow new concepts to be tried without losing any of the code already developed

By default it comes with a faily standard Unix style command line interface but there are many useful client interfaces such as WinCVS that provides a graphical user interface for most platforms (Windows/Unix/Linux/Mac) and CVSWeb that provided a browser based interface to view the repository. The WinCVS site also has links to other clients and other sites about CVS. A fully downloadable book can also be found at http://cvsbook.red-bean.com/

What does Bugzilla offer me ?

Bugzilla provides a simple web and e-mail based issue tracking system that allows mutiple products (or projects) and their components to have issues raised and prioritised and tracked through to resolution.

If features :

integrated, product-based granular security schema
inter-bug dependencies and dependency graphing
advanced reporting capabilities
a robust, stable RDBMS back-end
extensive configurability
a very well-understood and well-thought-out natural bug resolution protocol
email, XML, console, and HTTP APIs
available integration with automated software configuration management systems, including Perforce and CVS (through the Bugzilla email interface and checkin/checkout scripts)

So where is the catch ?

There isn’t one, all you need is a small Windows or Linux server that runs a web server such as IIS or Apache with enough disk to hold your repositories. Our 10,000 issue bugzilla database takes 5Mb and we have about 1Gb or source code from mutliple projects. A machine of entry level standard with a modern but not superfast processor, 0.5Gb memory and 100Gb hard disk and a backup device (recordable DVD or tape drive) with an OS of your choice will easily support most organisations.

You may need some help configuring the tools but the download and installation are easily followed from the various websites and you can always Google CVS or Google Bugzilla

What should we put into CVS ?

The CVS repository should hold everything you need to (re-)build your data warehouse this will include:

Requirement Specifications
Analysis and Design Documents
Code
Data Models (if you use a tool such as ErWin or Oracle Designer take an export of the repository and check it in after you have made any changes
Universes and End User Layers (if you use a tool such as Business Objects or Oracle Discoverer export the metadata for the business layer and report definitions)
System Configuration scripts (how the systems and databases that hold your data warehouse are built)
Manuals and user/system docuemntation
ETL Metadata exports
Any other metadata
Any other documents

What should we put into Bugzilla ?

The Bugzilla repository should hold everything you need to track issues, changes and enhancements to your data warehouse this will include:

All issues (prioritised via the tool)
All enhancement requests
All dependencies
Links to the CVS repository
Comments and observations about the issue
Issue status
Anything else that relates to a bugzilla feature

Conclusion

If your orgainsation has well organised procedures and development practices with existing tools, has a large budget to buy tools and external consultants to deploy them, or is so completely adverse to the fact that something available for free via open source could be useful then our plea in the second paragraph to ‘not stop reading … this advice is free’ was misguided.

If, however, your project:

has no budget for this type of facility but you’ve realised you desparately need it
has lost a day/week/month’s worth of work and don’t want to make the same mistake after you have recovered
has an enthusiastic techie who is willing to set it up and have a go
needs to prove it is in control of the development cycle to the project office/auditors
just wants to do the right thing

then hopefully this article has given you some pointers, and of course if you want some further help you can also contact us