Data Management & Warehousing
  • Welcome
    • Contact Us
    • QR Codes
    • Privacy & Cookie Policy
    • Internal
    • Websites
  • White Papers
  • Presentations
  • Blog
  • Our Services
    • Rates
Home » Posts tagged "Project Services"

Tag Archives: Project Services

Project Services HowTo

Posted on 8 April 2007 by David M. Walker Posted in Techniques Leave a comment

Objective

Project Services is an integration of software and doman knowledge. Data Management & Warehousing has an approach to building data warehouses. This approach includes a number of templates and white papers including approaches to architecture, governance and documentation.

In order to run a project using this approach it is useful to have a change control system, a task and issue tracking system, a wiki, etc. This document describes how to build the software aspects of the solution

Installed Software Solution

The objective is to build a system using the following packages:

Server

  • Subversion
    http://subversion.tigris.org/
  • Trac
    http://trac.edgewall.org/
  • ViewVC
    http://www.viewvc.org/
  • User_Manage
    http://stein.cshl.org/~lstein/user_manage/

Client

  • Tortoise SVN
    http://tortoisesvn.tigris.org/

Installed Software

The Data Management & Warehousing system used the following software

  • CentOS 4.5 fully up-to-date
  • Additional RPMs from http://dag.wieers.com/rpm/
    • subversion-perl-1.4.4-0.1.el4.rf
    • subversion-1.4.4-0.1.el4.rf
    • subversion-devel-1.4.4-0.1.el4.rf
    • python-clearsilver-0.10.4-2.el4.rf
    • clearsilver-devel-0.10.4-2.el4.rf
    • clearsilver-0.10.4-2.el4.rf
    • perl-ClearSilver-0.10.4-2.el4.rf
    • python-clearsilver-0.10.4-2.el4.rf
  • Source code downloads
    • trac-0.10.4.tar.gz
    • viewvc-1.0.4.tar.gz
    • user_manage.tar.gz

Naming conventions used

Wherever there is a name or path that needs to be set it is written in uppercase and should be substitued appropriately

CMS_PAGE: A location of a CMS managed page e.g. http://www.datamgmt.com/?PAGE=&47
DISK_ROOT: A location not visible to the webserver e.g. /hosting/datamgmt.com/projects
DOMAIN: The domain e.g. datamgmt.com
IPADDRESS: The IP Address e.g. 10.10.10.10
PORT: The IP port e.g. 80
SUBDOMAIN: The subdomain where the projects are hosted e.g. projects.datamgmt.com
WEB_ROOT: A location visible to the webserver e.g. /hosting/datamgmt.com/website/projects

Text prefixed with a hash ‘#’ it is a unix command to be typed.
Text in italics should be edited into the appropriate file

To set up an additional project the steps in red text need to be repeated for the new project.

Mail Alias

Create a e-mail alias for the project administrators that can be used in the configuration and directed as appropriate e.g.

projects-admin@DOMAIN

Create the Subversion Repository

# mkdir -p DISK_ROOT/PROJECT/svn/ 
# svnadmin create --fs-type fsfs DISK_ROOT/PROJECT/svn/

 

Install ViewVC

# mkdir DISK_ROOT/cgi-bin  
# cp /usr/local/viewvc-1.0.4/bin/cgi/viewvc.cgi DISK_ROOT/cgi-bin

Then update the /usr/local/viewvc-1.0.4/viewvc.conf file with the following lines:

#cvs_roots 
svn_roots = PROJECT: DISK_ROOT/PROJECT/svn
default_root = svn
address = Projects Administration

 

Install user_manage

# cp user_manage WEB_ROOT/cgi-bin

Edit the user_manage script as follows:

$CONFIG_FILE = 'DISK_ROOT/etc/realms.conf';
# mkdir DISK_ROOT/etc

Create and edit the file DISK_ROOT/etc/realms.conf:

<Realm projects.datamgmt.com/PROJECT>
        Users      DISK_ROOT/PROJECT/etc/passwd.http
        Groups     DISK_ROOT/PROJECT/etc/group.http
        Type       Text
</Realm>
# cd DISK_ROOT/PROJECT/etc

# htpasswd -cb passwd.http USERNAME PASSWORD

Create and edit the file DISK_ROOT/PROJECT/etc/group.http

administrators: USERNAME
users: USERNAME

 

Configure Trac

# mkdir DISK_ROOT/tmp
# chown apache.apache DISK_ROOT/tmp
# chmod 775 DISK_ROOT/tmp
# trac-admin DISK_ROOT/PROJECT/trac initenv

Accept default values except

Project Name [My Project]> PROJECT 
Path to repository [/path/to/repos]> DISK_ROOT/PROJECT/svn

Edit the configuration file DISK_ROOT/PROJECT/trac/conf/trac.ini

[header_logo] 
 link = http://SUBDOMAIN 
 src = /img/yourlogo.png 
[notification] 
 always_notify_owner = true 
 always_notify_reporter = true 
 smtp_default_domain = DOMAIN 
 smtp_from = projects-admin@DOMAIN 
 smtp_replyto = projects-admin@DOMAIN 
 smtp_enabled = true 
[project] 
 descr = PROJECT 
 name = PROJECT 
 url = http://SUBDOMAIN/PROJECT/trac 
[trac] 
 base_url = http://SUBDOMAIN/PROJECT/trac

 

Configure .htaccess files

Add the following .htaccess files

Redirect the DOMAIN homepage to the main website Content Management System page for requests (This rule re-directs requests / and index.html requests)

Create and edit WEB_ROOT/.htaccess

RewriteEngine on RewriteRule ^$ CMS_PAGE RewriteRule ^index.html$ CMS_PAGE 

Redirect the DOMAIN/PROJECT homepage to the Trac homepage (This rule re-directs requests / and and index.html requests)

Create and edit WEB_ROOT/PROJECT/.htaccess

RewriteEngine on RewriteRule ^$ http://DOMAIN/PROJECT/trac RewriteRule ^index.html$ http://DOMAIN/PROJECT/trac 

 

Set up Trac security

# trac-admin DISK_ROOT/PROJECT/trac permission list 
# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME user 
# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME admin 
# trac-admin DISK_ROOT/PROJECT/trac permission add user TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY 
# trac-admin DISK_ROOT/PROJECT/trac permission add admin CONFIG_VIEW MILESTONE_ADMIN REPORT_ADMIN ROADMAP_ADMIN TICKET_ADMIN TRAC_ADMIN WIKI_ADMIN 
# trac-admin DISK_ROOT/PROJECT/trac permission remove anonymous TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY

Modify other permissions to suit individual requirements

Modifications to the httpd.conf file

<VirtualHost  IPADDRESS:PORT>

ServerAdmin   projects-admin@DOMAIN
DocumentRoot  WEB_ROOT
ServerName    SUBDOMAIN
ErrorLog      WEB_ROOT/logs/errors.log
CustomLog     WEB_ROOT/logs/access.log combined env=!DontLog
ScriptAlias   /cgi-bin  WEB_ROOT/cgi-bin

   AllowOverride All

   Options ExecCGI FollowSymLinks

<Directory WEB_ROOT/PROJECT> 
 AuthType Basic 
 AuthName "SUBDOMAIN/PROJECT" 
 AuthUserFile DISK_ROOT/PROJECT/etc/passwd.http 
 AuthGroupFile DISK_ROOT/PROJECT/etc/group.http 
 Require valid-user 
 AllowOverride All 
</Directory> 
<Location /PROJECT/trac> 
 SetHandler mod_python 
 PythonInterpreter main_interpreter 
 PythonHandler trac.web.modpython_frontend 
 PythonOption TracEnv DISK_ROOT/PROJECT/trac 
 PythonOption TracUriRoot /PROJECT/trac 
 SetEnv PYTHON_EGG_CACHE DISK_ROOT/tmp 
</Location> 
<Location /PROJECT/repository> 
 DAV svn 
 SVNParentPath DISK_ROOT 
</Location>

</VirtualHost>

 

Completing the installation and testing

After all of the above is done the web server needs to be restarted

# apachectl stop
# apachectl start

http://SUBDOMAIN
This should take you to the CMS page that acts as an introduction to all the projects

http://SUBDOMAIN/cgi-bin/user_manage?realm=SUBDOMAIN/PROJECT
This should take you to the user management page of the sub-domain and project.

http://SUBDOMAIN/PROJECT
This should take you to the wiki page that acts as the front page of the individual project

http://SUBDOMAIN/PROJECT/cgi-bin/viewvc.cgi
This should take you to the ViewVC page the views the source code repository

Other useful utilities

There are a number of useful plugins:

http://trac-hacks.org/wiki

Tip: Create a directory (e.g. /tmp/trac_plugins) and carry out the work from there.

Tip: Install the plugins globally using the –prefix=/usr (see below)

Tip: If you are installing a second project copy the trac.ini over after doing all the installs and then simply search and replace the project name instead of doing all the edits.

Tip: Once the ini admin module is installed much of the enabling etc. can be done from here.

Tip: Sometimes after installing a new plugin it does not appear. Try restarting the browser and webserver as stuff might be cached.

Tip: Sometimes the following line
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
can be added to the file /usr/lib/python2.3/site-packages/easy-install.pth causing a Server 500 error. This should be commented out

EZ Setup Utility

# wget http://peak.telecommunity.com/dist/ez_setup.py 
# python ez_setup.py --prefix=/usr

 

General Plugin Installation

# svn co http://PLUGIN_SVN_URL
# cd PLUGIN/(0.10)
# python setup.py egg_info
# python setup.py bdist_egg
# cd dist
# easy_install –prefix=/usr *.egg

Edit DISK_ROOT/PROJECT/conf/trac.ini

[components]
PLUGIN.* = enabled

Implemented Plugins

Account Manager

ini Admin

Master Tickets
Also note that the following patch must be applied before building

Nav Add

Rev Tree
Tried to install but have had issues so abandoned

SVN Authz Admin

Test Case Management

TracNav

Web Admin

Other useful plugins not installed

Timing and Estimation

Work Log

Issues

Sometimes after installing a plugin the server issues an error

Editing:

/usr/lib/python2.3/site-packages/easy-install.pth

and commenting out the line:

#import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Appears to fix this

Configuration Management Issue Tracking Project Services SVN Trac

Project Services

Posted on 8 March 2007 by David M. Walker Posted in Techniques Leave a comment

Welcome to Project Services

In the Data Management & Warehousing White Paper Data Warehouse Governance the importance of change control and issue tracking to a successful data warehousing project are highlighted. This paper also highlights the importance of a lightweight or agile development methodology and how using a wiki can help improve the develpment process.

Project Services offers the following:

  • A demonstration of a complete, consistent and co-herent set of tools for project planning, task, risk, issue and enhancement tracking, document and source code control, test management and project wiki integrated together to support a data warehouse development
  • A free hosted environment for consultancy clients of Data Management & Warehousing on which to run their project
  • A low cost hosted environment for projects not using Data Management & Warehousing consultancy services but wishing to use our approach to developing successful data warehouses.
  • A guide for those wishing to implement their own data warehouse project governance.

 

What is the Project Services platform ?

Each project gets it’s own private area in which to operate. No data is shared between projects. For example the ‘demo’ project can be found at http://projects.datamgmt.com/demo

Each project has the following features;

  • Subversion:
    An open-source revision control system
  • Trac:
    A web-based software project management and bug/issue tracking system that also provides an interface to Subversion and an integrated wiki
  • User Management:
    A web-based interface to allow management of usernames, groups and passwords.
  • Tortoise SVN:
    A client for Subversion that Subversion commands directly in Windows explorer.

Note: Since this service works over http or https it will work from within an corporate firewall

Can I see the demo ?

The demostration system is available at http://projects.datamgmt.com/demo with a username of ‘guest’ and a password of ‘guest’. At the moment there is not much available becuase it will take some time to build up a set of worthwhile examples, however we are actively working on it so please check back regularly.

If you are interested in a hosted environment the please Contact Us

Why the change of platform ?

For many years Data Management & Warehousing has recommended a cheap open source solution to this problem by implementing Bugzilla and CVS.

Whilst Bugzilla is web based, CVS normally required an SSH connection and the solution also required the use and integration of a seperate Wiki solution. Using Subversion and Trac has allowed Data Management & Warehousing to implement a fully http based solution which integrated change control, task and issue management and a project wiki in a single solution, but with all the heritage of the previous tools.

How can we put Project Services on our own server ?

All the code used to create Project Services is Open Source. Therefore anyone can build it on their own server. For our own internal documentation and in case anyone else is interested we have created the Project Services HowTo

Agile Configuration Management Data Warehousing Governance Issue Tracking Project Management Project Services SVN Trac

Configuration Management & Issue Tracking

Posted on 2 June 2004 by David M. Walker Posted in Techniques Leave a comment
One commonly overlooked side of building a data warehouse is the need for configuration management and issue tracking. This article looks at a way to do this.
Since this article was written we have upgraded from Bugzilla to Trac and CVS to SVN and integrated in in our Project Services

Introduction

ETL tool transformations, data models, query tools & report writers, analysis documents and data quality reports are all part of the collection of information gathered during the build of a data warehouse. Once in production the tracking of issues that arise from day to day operations and the relationship to the ETL process also need tracking.

As a consultancy we normally steer clear of recommending specific products as generic solutions and only do so for particular clients who need tools to meet specific situations. However we have generally noted most organisations do not do the configuration management and issue tracking well and that commercial solutions, unless part of an existing corporate infrastructure, are costly to deploy.

If this is normally so expensive what should we do ?

The premise of this article is using open source software. Don’t stop reading – like the software this advice is free so it is worth a look.

Whatever is wrong with individual pieces of open source software the one thing that has worked well is the issue tracking and source code control. Sites such as http://sourceforge.net have over 82,000 projects and 858,000 users (as at 1st June 2004).

There are two pieces of software that stand out and are widely used for configuration management and issue tracking. These are:

  • CVS: CVS is a version control system and important component of Source Configuration Management (SCM) that records the history of sources files and documents.
  • Bugzilla: Bugzilla is one example of a class of programs called “Defect Tracking Systems”, or, more commonly, “Bug-Tracking Systems” that groups of developers to keep track of outstanding bugs in their product effectively.

What does CVS offer me ?

CVS provides a repository on a server (Windows or Linux – see later) that can store and version files (including binary files and special format files such as Microsoft Word). When you create a file you ‘check it in’ and when you want to read it again or deploy it you ‘check it out’. If you want to edit the file then you check it out in a locked mode to prevent others editing it whist you are. It is also possible mutliple developers to check out the code, work on it and check it back in and ask for conflict resolution etc. Finally you can tag ‘releases’ and have code branhes off the main development that allow new concepts to be tried without losing any of the code already developed

By default it comes with a faily standard Unix style command line interface but there are many useful client interfaces such as WinCVS that provides a graphical user interface for most platforms (Windows/Unix/Linux/Mac) and CVSWeb that provided a browser based interface to view the repository. The WinCVS site also has links to other clients and other sites about CVS. A fully downloadable book can also be found at http://cvsbook.red-bean.com/

What does Bugzilla offer me ?

Bugzilla provides a simple web and e-mail based issue tracking system that allows mutiple products (or projects) and their components to have issues raised and prioritised and tracked through to resolution.

If features :

  • integrated, product-based granular security schema
  • inter-bug dependencies and dependency graphing
  • advanced reporting capabilities
  • a robust, stable RDBMS back-end
  • extensive configurability
  • a very well-understood and well-thought-out natural bug resolution protocol
  • email, XML, console, and HTTP APIs
  • available integration with automated software configuration management systems, including Perforce and CVS (through the Bugzilla email interface and checkin/checkout scripts)

So where is the catch ?

There isn’t one, all you need is a small Windows or Linux server that runs a web server such as IIS or Apache with enough disk to hold your repositories. Our 10,000 issue bugzilla database takes 5Mb and we have about 1Gb or source code from mutliple projects. A machine of entry level standard with a modern but not superfast processor, 0.5Gb memory and 100Gb hard disk and a backup device (recordable DVD or tape drive) with an OS of your choice will easily support most organisations.

You may need some help configuring the tools but the download and installation are easily followed from the various websites and you can always Google CVS or Google Bugzilla

What should we put into CVS ?

The CVS repository should hold everything you need to (re-)build your data warehouse this will include:

  • Requirement Specifications
  • Analysis and Design Documents
  • Code
  • Data Models (if you use a tool such as ErWin or Oracle Designer take an export of the repository and check it in after you have made any changes
  • Universes and End User Layers (if you use a tool such as Business Objects or Oracle Discoverer export the metadata for the business layer and report definitions)
  • System Configuration scripts (how the systems and databases that hold your data warehouse are built)
  • Manuals and user/system docuemntation
  • ETL Metadata exports
  • Any other metadata
  • Any other documents

What should we put into Bugzilla ?

The Bugzilla repository should hold everything you need to track issues, changes and enhancements to your data warehouse this will include:

  • All issues (prioritised via the tool)
  • All enhancement requests
  • All dependencies
  • Links to the CVS repository
  • Comments and observations about the issue
  • Issue status
  • Anything else that relates to a bugzilla feature

Conclusion

If your orgainsation has well organised procedures and development practices with existing tools, has a large budget to buy tools and external consultants to deploy them, or is so completely adverse to the fact that something available for free via open source could be useful then our plea in the second paragraph to ‘not stop reading … this advice is free’ was misguided.

If, however, your project:

  • has no budget for this type of facility but you’ve realised you desparately need it
  • has lost a day/week/month’s worth of work and don’t want to make the same mistake after you have recovered
  • has an enthusiastic techie who is willing to set it up and have a go
  • needs to prove it is in control of the development cycle to the project office/auditors
  • just wants to do the right thing

then hopefully this article has given you some pointers, and of course if you want some further help you can also contact us

Bugzilla Configuration Management CVS ETL Issue Tracking Project Management Project Services SVN Trac

Archives

Random Quote

Find the truth. Act on it.

— Dan Brown, Deception Point

Contact Details

Tel (UK): 01458 850 433
Skype : datamgmt

Registered Office

Manchester House, High Street
Stalbridge, Sturminster Newton
Dorset, DT10 2LL
United Kingdom

Company Registration

Registered in England and Wales
Registration Number: 3526504
VAT Registration: GB724448236
Directors: DM & HJ Walker

© 1995-2013 Data Management & Warehousing
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.Accept Read More
Privacy & Cookies Policy