An ETIS presentation from Brussels in March 2013
This presentation looks at what SaaS and Cloud mean for BI and analytics in Telcos.
Read more in the presentation …
Download BI SaaS & Cloud Strategies for Telcos now
I have been aware for a few months of the Slow Web Movement, which, like other parts of the slow movement, advocates a cultural shift toward slowing down life’s pace. More specifically it is about getting more value and less volume from the web, getting quality information when it is appropriate and not being overwhelmed with a constant flood of real-time reports. Reading Jack Cheng’s Slow Web and Joshua Gross’ The Future Is Not Real Time blog posts prompted me to add these thoughts.
Firstly a story: I was delivering a business intelligence solution for a large European Telco and one of the ‘key requirements’ was to have ‘near real-time’ reporting of the response to adverts being run on television. The specific requirement from the marketing director was to know in real-time how many people were prompted to ring a call centre after an advert was screened. The challenge I laid down to him was “What are you going to do with this information? You are sitting at home at 10pm and the advert shows on TV, your smart phone buzzes five minutes later and tells you that 50 people have rung into the call centre using the promotion code.” The conversation wasn’t a smooth one – “I understand marketing – do you?” but I pushed “Was he going to ring up the TV station and get them to switch to an alternate advert if the numbers were higher/lower than he wanted? Or perhaps he was going to ring the manager of the call centre and get them to bring in more staff or release some staff?” When I pressed he saw my point – he was going to get the information and then review it with his staff over the coming weeks to plan future campaigns. What he didn’t admit was that to some extent having real-time reports was about prestige among his peers in the organisation.
The cost of delivering his requirement would have been huge – developing business intelligence is still a costly exercise and the overhead of introducing real-time requirements is massive. There are genuine useful examples where real-time analysis is vital but treat them as special, don’t build your entire business intelligence environment to support it. And don’t allow a system to be built for prestige – ask what the value is.
Of course this concept isn’t new – in March 2006 I presented From Volume to Value: What Next Generation Telco Data Warehouses Must Do to Provide Value to the Business which looked at reducing the volume of information produced in favour of delivering more value with that information.
One of the enduring ideas from this presentation is the use of RSS as a delivery mechanism for Business Intelligence. Building RSS feeds is easy, and that includes creating feeds with attachments. What is more – the ability to read an RSS feeds is almost ubiquitous – you’ll find RSS readers on you Windows, Linux or OS X computer as well as on your Android and iOS handset and many other places besides. RSS has a rich optional metadata set that can optionally include categorisations, publication times, authors and many other elements that allow a user to effectively filter what they need. A metadata rich, RSS feed that uses PDF attachments creates a widespread distribution mechanism that greatly reduces the need for software licences for managers that will rarely log into expensive desktop tools. It puts the information directly into the users hands and allows them to handle it when they want to or need to.
Good business intelligence is about supporting business users to make good business decisions – it is not a technological end in itself. Sometimes a tortoise with timely information is going to perform better overall than a hare that shots off at the start and has to re-assess further down the line – just like in Aesop’s Fable
This brief blog post is a summary of the various things we have found on the upgrades of our Apple Mac infrastructure. It is primarily intended for internal use however it may prove useful to others. Some of this information is widely available and was prior to the release of Mountain Lion.
MacBooks from 2007 won’t upgrade
All the black Macbooks from 2007 and before won’t upgrade as they are only 32-bit and Mountain Lion requires 64-bit. You will need a newer machine to run Mountain Lion
Check your disk before you start
Use Disk Utility to Repair Disk Permissions and Repair Disk before you start
Some MacBook Pros have problems with upgrades
This article and this article discuss the fact that some MacBook Pros seem to have a hard disk crash during the upgrade process. It is not really a hard disk crash but a problem with the type of partition table on the disk. You need to have a GUID partition table and some models don’t have one. Firstly make sure you have a backup. Secondly if you have a large enough external drive take a second disk image using Disk Utility just to make sure. If your machine upgrade fails then reformat the disk using a GUID partition – this has to be done in recovery mode. Recover the disk, restart and re-run the upgrade and you should be on your way.
SVN (Subversion) is not installed in /usr/bin as before
You now need to have XCode installed to get SVN. Once you installed XCode via the AppStore you can open a terminal session and then do this:
sudo su –
ln -s /Applications/Xcode.app/
This creates a symbolic link and now command line SVN will work again as expected – thanks to this post
Messages can be used with Facebook Chat
Basically add a Jabber account with your username and password from Facebook and the server set to chat.facebook.com on port 5222 – thanks to this post
Gatekeeper and downloaded software
A new option exists to protect against malicious software downloads. If you download something from somewhere other than the App Store you might get a warning message that prohibits the downloaded program from running. This can be circumvented by going to System Preferences -> Security & Privacy -> General where you can set the default level. If you just want to run any other downloaded app or utility, you can control-click the app, choose Open from the pop-up menu, and confirm that you want to approve the app for your system; you only have to give this approval once. More here
Software that doesn’t work
I needed to get Ruby-on-Rails up to date and also had wanted to switch to Brew as a package manager so I used these instructions on get XCode etc. up to date and followed the Brew Instructions. It is really important to clean up the system with the ‘brew doctor‘ command
The upgrade has been fairly clean, a few pieces of software don’t run because they are not 64-bit compatible but nothing critical. If we find anything else we will upgrade this post
A data warehouse project will usually involve many pieces of software.
These can be broken into two categories – architectural software and supporting tools. The architectural software will include the database, ETL tool, reporting tools, etc. but in this post I am going to look at the supporting tools we use.
I should say from the outset that by choice we have a shared Linux server and that the team like to use Apple Macs rather than Microsoft Windows machines and so our tools reflect the platforms we use. We often work inside organisations that are completely Microsoft Windows-based and so we are careful not to work in any way that would preclude interoperability with our clients environments.
On the server we need to have the following features
To support this we use Subversion (also known as SVN) for version control and Trac which provides the ticketing and wiki functionality, both are open source projects and so are free. We configure them using our Project Services guidelines. Trac requires a server side database for which we normally use MySQL and phpMyAdmin to manage the MySQL database.
There are some hosted services that are quite useful
On our desktops we have a range of software
As ‘root’ on your system perform the following four commands (Note: Version numbers may vary)
To install R
rpm -ivh http://mirror.chpc.utah.edu/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
yum install R
To install RStudio
rpm -Uvh rstudio-server-0.96.316-i686.rpm
Once you have done this (and assuming you don’t need to open any firewalls etc.) you can simply use the URL in your web browser.
The username and password will be the same as you would use to log into the server that R and RStudio are hosted on.
To update R as ‘root’ on your system simply type
yum update R
To update RStudio simply repeat the steps above – but changing the version numbers
You can also find other useful R resources at: http://www.r-bloggers.com
So your manager wants reports with sexy graphs – what are your options and what are the pros and cons
I have found eight usable packages with different considerations
This is a series of libraries including one for Ruby that allow the generation of great looking charts that are very dynamic (see the demo on the website). Data is easily passed via JSON making it easy to integrate. The only downside is that it requires the client to have an Adobe Flash player installed. This is common enough for it not to be a problem for most people but it does limit the portability. Also as it generates a flash object rather than a JPG, PNG or GIF it means that this is suitable for online output but not useful in RSS feeds etc and when printed all the dynamic features are lost anyway.
Gruff provides a simple and easy to use Ruby package that works well with Rails. It does require that ImageMagick/RMagick is installed on the server (HowTo for OSX) and this can be a pain for some people. The advantage is that it produces good looking, native SVG, PNG, JPG, GIF files so they are easily portable and printable
Scruffy is very similar to Gruff although appears to be less well used and not recently updated. It too requires ImageMagick/RMagick installed and whilst it is based on SVG is capable of producing other graph formats.
Sparklines are small graphs that are usually embedded within the text of a paragraph or placed inside tables. The standard output is a png file and it also requires ImageMagick/RMagick to be installed. This size of the graphs makes it useful for small graphs such as indicators rather than main report graphics.
Google charts provide perhaps the easiest interface to creating a graph. All the parameters get bundled up in the URL and Google does all the work. Added to this is gchartrb that provides an easy and intuitive way to build the URL. All this means that there is nothing to install on the server side however it also means that to get access to your image you have to be connected to the internet and pass data which your company may regard as secure to the Google servers. This may make it impractical for use with BI data.
SVG:::Graph is a pure Ruby library for generating charts, which are a type of graph where the values of one axis are not scalar. The output is an SVG which might not be supported in all browsers or a PNG file
A Ruby on Rails helper for making simple graphs. The graphs use only CSS/HTML (and an optional gradient image). Great at creating quick simple online graphs but very limited in its capabilities
If you are looking for a simple graph that is easy to generate then gchartrb is for you
If you are looking for dynamic and very attractive content then Flot or OpenFlashChart are your options
If you are looking for small KPI indicator type graphs consider Sparklines
If you are looking for good looking images that can be distributed consider Gruff, Scruffy and SVG::Graph
If you are happy with a quick and easy CSS based solution then look at CSS Graph Helper
This article was originally published on BIonRails, another Data Management & Warehousing website
Project Services is an integration of software and doman knowledge. Data Management & Warehousing has an approach to building data warehouses. This approach includes a number of templates and white papers including approaches to architecture, governance and documentation.
In order to run a project using this approach it is useful to have a change control system, a task and issue tracking system, a wiki, etc. This document describes how to build the software aspects of the solution
The objective is to build a system using the following packages:
The Data Management & Warehousing system used the following software
Wherever there is a name or path that needs to be set it is written in uppercase and should be substitued appropriately
CMS_PAGE: A location of a CMS managed page e.g. http://www.datamgmt.com/?PAGE=&47
DISK_ROOT: A location not visible to the webserver e.g. /hosting/datamgmt.com/projects
DOMAIN: The domain e.g. datamgmt.com
IPADDRESS: The IP Address e.g. 10.10.10.10
PORT: The IP port e.g. 80
SUBDOMAIN: The subdomain where the projects are hosted e.g. projects.datamgmt.com
WEB_ROOT: A location visible to the webserver e.g. /hosting/datamgmt.com/website/projects
Text prefixed with a hash ‘#’ it is a unix command to be typed.
Text in italics should be edited into the appropriate file
To set up an additional project the steps in red text need to be repeated for the new project.
Create a e-mail alias for the project administrators that can be used in the configuration and directed as appropriate e.g.
# mkdir -p DISK_ROOT/PROJECT/svn/# svnadmin create --fs-type fsfs DISK_ROOT/PROJECT/svn/
# mkdir DISK_ROOT/cgi-bin # cp /usr/local/viewvc-1.0.4/bin/cgi/viewvc.cgi DISK_ROOT/cgi-bin
Then update the /usr/local/viewvc-1.0.4/viewvc.conf file with the following lines:
#cvs_roots svn_roots = PROJECT: DISK_ROOT/PROJECT/svn default_root = svn address = Projects Administration
# cp user_manage WEB_ROOT/cgi-bin
Edit the user_manage script as follows:
$CONFIG_FILE = 'DISK_ROOT/etc/realms.conf';
# mkdir DISK_ROOT/etc
Create and edit the file DISK_ROOT/etc/realms.conf:
<Realm projects.datamgmt.com/PROJECT> Users DISK_ROOT/PROJECT/etc/passwd.http Groups DISK_ROOT/PROJECT/etc/group.http Type Text </Realm># cd DISK_ROOT/PROJECT/etc # htpasswd -cb passwd.http USERNAME PASSWORD
Create and edit the file DISK_ROOT/PROJECT/etc/group.http
administrators: USERNAME users: USERNAME
# mkdir DISK_ROOT/tmp # chown apache.apache DISK_ROOT/tmp # chmod 775 DISK_ROOT/tmp# trac-admin DISK_ROOT/PROJECT/trac initenv
Accept default values except
Project Name [My Project]> PROJECTPath to repository [/path/to/repos]> DISK_ROOT/PROJECT/svn
Edit the configuration file DISK_ROOT/PROJECT/trac/conf/trac.ini
[header_logo]link = http://SUBDOMAINsrc = /img/yourlogo.png[notification]always_notify_owner = truealways_notify_reporter = truesmtp_default_domain = DOMAINsmtp_from = projects-admin@DOMAINsmtp_replyto = projects-admin@DOMAINsmtp_enabled = true[project]descr = PROJECTname = PROJECTurl = http://SUBDOMAIN/PROJECT/trac[trac]base_url = http://SUBDOMAIN/PROJECT/trac
Add the following .htaccess files
Redirect the DOMAIN homepage to the main website Content Management System page for requests (This rule re-directs requests / and index.html requests)
Create and edit WEB_ROOT/.htaccess
RewriteEngine on RewriteRule ^$ CMS_PAGE RewriteRule ^index.html$ CMS_PAGE
Redirect the DOMAIN/PROJECT homepage to the Trac homepage (This rule re-directs requests / and and index.html requests)
Create and edit WEB_ROOT/PROJECT/.htaccess
RewriteEngine on RewriteRule ^$ http://DOMAIN/PROJECT/trac RewriteRule ^index.html$ http://DOMAIN/PROJECT/trac
# trac-admin DISK_ROOT/PROJECT/trac permission list# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME user# trac-admin DISK_ROOT/PROJECT/trac permission add USERNAME admin# trac-admin DISK_ROOT/PROJECT/trac permission add user TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY# trac-admin DISK_ROOT/PROJECT/trac permission add admin CONFIG_VIEW MILESTONE_ADMIN REPORT_ADMIN ROADMAP_ADMIN TICKET_ADMIN TRAC_ADMIN WIKI_ADMIN# trac-admin DISK_ROOT/PROJECT/trac permission remove anonymous TICKET_CREATE TICKET_MODIFY WIKI_CREATE WIKI_MODIFY
Modify other permissions to suit individual requirements
<VirtualHost IPADDRESS:PORT> ServerAdmin projects-admin@DOMAIN DocumentRoot WEB_ROOT ServerName SUBDOMAIN ErrorLog WEB_ROOT/logs/errors.log CustomLog WEB_ROOT/logs/access.log combined env=!DontLog ScriptAlias /cgi-bin WEB_ROOT/cgi-bin AllowOverride All Options ExecCGI FollowSymLinks <Directory WEB_ROOT/PROJECT>AuthType BasicAuthName "SUBDOMAIN/PROJECT"AuthUserFile DISK_ROOT/PROJECT/etc/passwd.httpAuthGroupFile DISK_ROOT/PROJECT/etc/group.httpRequire valid-userAllowOverride All</Directory><Location /PROJECT/trac>SetHandler mod_pythonPythonInterpreter main_interpreterPythonHandler trac.web.modpython_frontendPythonOption TracEnv DISK_ROOT/PROJECT/tracPythonOption TracUriRoot /PROJECT/tracSetEnv PYTHON_EGG_CACHE DISK_ROOT/tmp</Location><Location /PROJECT/repository>DAV svnSVNParentPath DISK_ROOT</Location> </VirtualHost>
After all of the above is done the web server needs to be restarted
# apachectl stop # apachectl start
This should take you to the CMS page that acts as an introduction to all the projects
This should take you to the user management page of the sub-domain and project.
This should take you to the wiki page that acts as the front page of the individual project
This should take you to the ViewVC page the views the source code repository
There are a number of useful plugins:
Tip: Create a directory (e.g. /tmp/trac_plugins) and carry out the work from there.
Tip: Install the plugins globally using the –prefix=/usr (see below)
Tip: If you are installing a second project copy the trac.ini over after doing all the installs and then simply search and replace the project name instead of doing all the edits.
Tip: Once the ini admin module is installed much of the enabling etc. can be done from here.
Tip: Sometimes after installing a new plugin it does not appear. Try restarting the browser and webserver as stuff might be cached.
Tip: Sometimes the following line
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
can be added to the file /usr/lib/python2.3/site-packages/easy-install.pth causing a Server 500 error. This should be commented out
# wget http://peak.telecommunity.com/dist/ez_setup.py # python ez_setup.py --prefix=/usr
# svn co http://PLUGIN_SVN_URL
# cd PLUGIN/(0.10)
# python setup.py egg_info
# python setup.py bdist_egg
# cd dist
# easy_install –prefix=/usr *.egg
PLUGIN.* = enabled
Tried to install but have had issues so abandoned
Sometimes after installing a plugin the server issues an error
and commenting out the line:
#import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,’__egginsert’,0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
Appears to fix this
Microsoft BGInfo (formally SysInternals)
One of the most commonly used utilities on Windows platforms is BG Info. It creates a replacement screen background with key information about the system configuration. BG Info provides a simple interface to select the items that you want which is very useful either on large blade configurations or using virtual machines (e.g. when using VMWare)
One limitation is the information shown when a system has multiple network cards. Fortunately BGInfo allows the user to configure a local VBScript to run anything the system administrator or user wants. Data Management & Warehousing have created a script that correctly displays the network information for all active adaptors in the system.
Configuring the script
For the purposes of this document we have installed the BGInfo software in C:\BGInfo. If your installation uses a different location you should adjust this accordingly.
For quick configuration you can copy the two files (right-click save Config.bgi and network.vbs) into C:\BGInfo and then start BGInfo and open the file Config.bgi and make any more changes you need, save it and execute it.
To refresh the information at startup create a shortcut in the ‘C:\Documents and Settings\All Users\Start Menu\Programs\Startup’ directory where the target is set to:
C:\BGInfo\BGInfo.exe C:\BGInfo\Config.bgi /timer:0
For manual configuration or if you make any changes to the VBS script you must follow the instructions below. It is important that this is done after every change to the network.vbs script otherwise the changes will not be included.
Once you have opened up BGInfo click on the countdown timer on the top right to stop it from automatically closing the window. and then click on the ‘Custom’ button on the right hand side of the window in the ‘Fields’ section.
If this is the first time that you have created a User Defined Field then you must click on ‘New…’ otherwise highlight ‘Network Info’ and click on ‘Edit…’ followed by OK.
Now make sure that the Identifier field says ‘Network Info’, check the VB Script file option and use the Browse button to select the network.vbs script. Once you have done all this click on the ‘OK’ button to close this screen.
This will return you to the user defined fields window and again you must click on OK to return to the main window.
Once you are in the main window you can add Network Info as a field in the display by clicking on ‘Add’ and positioning item as required and then either click on ‘Apply’ to see how it looks or OK to make the change permanent.
Once this is done your screen will look something like the ones shown on this page. You can adjust background and foreground colours and images, position, etc. by following the instructions for the product to get the effect you want.
In the Data Management & Warehousing White Paper Data Warehouse Governance the importance of change control and issue tracking to a successful data warehousing project are highlighted. This paper also highlights the importance of a lightweight or agile development methodology and how using a wiki can help improve the develpment process.
Project Services offers the following:
Each project gets it’s own private area in which to operate. No data is shared between projects. For example the ‘demo’ project can be found at http://projects.datamgmt.com/demo
Each project has the following features;
Note: Since this service works over http or https it will work from within an corporate firewall
The demostration system is available at http://projects.datamgmt.com/demo with a username of ‘guest’ and a password of ‘guest’. At the moment there is not much available becuase it will take some time to build up a set of worthwhile examples, however we are actively working on it so please check back regularly.
If you are interested in a hosted environment the please Contact Us
Whilst Bugzilla is web based, CVS normally required an SSH connection and the solution also required the use and integration of a seperate Wiki solution. Using Subversion and Trac has allowed Data Management & Warehousing to implement a fully http based solution which integrated change control, task and issue management and a project wiki in a single solution, but with all the heritage of the previous tools.
All the code used to create Project Services is Open Source. Therefore anyone can build it on their own server. For our own internal documentation and in case anyone else is interested we have created the Project Services HowTo
To access and work with the the CVS repositories you must install a CVS client on your Windows PC that can talk to the CVS servers. There are scores of CVS clients available, but at a minimum you must install the CVS command line client.
This document also includes the instructions needed to install two GUI Windows clients for CVS: WinCvs and TortoiseCVS.
Use the instructions in this document to install the CVS command-line client and (optionally) one or both of the GUI clients on a Windows PC.
There are two documents that can help you get started with WinCvs:
The “WinCvs Daily Use Guide” is probably the best place to start.
TortoiseCVS is a plug-in for the Windows Explorer. It gives you easy, direct access to CVS commands from the Explorer. Even with WinCvs installed you will still probably find TortoiseCVS handy to have installed on your PC.
You can use WinCvs and TortoiseCvs (and the CVS command-line) interchangeably. For example, you can checkout a project using TortoiseCvs, work on it with WinCvs and then commit it using the CVS command-line.
As a rule this should also hold true for any other CVS clients you might decide to install on your PC. Possible exception: clients built into IDEs like Forte and Eclipse.
This page is based on an internal page written by Bruce Files at MPCT (later Aleri) in Chicago. It has been modified to make it a generic guide suitable for publication on the internet.