Data Management & Warehousing

Tag: SQL

  • Our Toolbox – Essential Software

    —

    by

    macleanwalker
    in

    A data warehouse project will usually involve many pieces of software. These can be broken into two categories – architectural software and supporting tools. The architectural software will include the database, ETL tool, reporting tools, etc. but in this post I am going to look at the supporting tools we use. I should say from…

  • Our Software

    —

    by

    macleanwalker
    in

    Over the years we have produced several utilities and open source projects. List list provides some information about their status GROUT – ETL tool including DAP & SQLUnload written in Korn Shell, C & PL/SQL – superseded by JST – code available on request JST – ETL tool written in Bash Shell – superseded by…

  • Oracle DBA Crib Sheet

    —

    by

    macleanwalker
    in

    Introduction The Oracle DBA Crib Sheet is not an exhastive list of what a DBA has to do, instead it has been written for a particular client whose support team occasionally need dba commands and don’t have them written down any where else. It features the commands that we have been asked to put in…

  • Data Transformation – Procedural & Non Procedural Solutions

    —

    by

    macleanwalker
    in

    This paper looks at a somewhat awkward data transformation, and at solutions written in SQL and in a procedural language. It describes some techniques which can be used to develop the solution in both languages. It also compares the solutions in terms of ease of development, performance and cost of maintenance. Are we building transformations…

  • Basic Data Quality Checks

    —

    by

    macleanwalker
    in

    This article looks at basic data quality audit that can be done within a database. Examples are given using Oracle syntax however the techniques can also be applied to other databases   Introduction The following article discusses some of the data quality issues that can be addressed by manual scripts on a copy of the…

  • Detecting Changed Data

    —

    by

    macleanwalker
    in

    Introduction When loading data warehouses, it is usually possible to decreases the load time very significantly by processing only changes since the last load, rather than completely refreshing all the data every time. This article describes one approach for detecting changes, which has been used successfully in a number of data warehouse projects. Background There…

  • Auditing Data Cleaning Updates

    —

    by

    macleanwalker
    in

    How to track what has been updated by data cleaning processes The Problem A common problem when building a data warehouse is to track and audit which records have had data quality updates or cleaning applied to them. A simple method is to add a bitmap that reflects the data quality updates have been applied.…