I usually tell people that data warehouses sound like a good idea and that every company should have one. People usually thank me politely for this but also advise me that they already have one – in fact they have around 17.
At that point, we agree that if you use a cowboy based Agile approach with no architecture you will add a lot of access databases and tactical databases to the existing chaos without simplifying the environment. So we agree that we should do a substantial amount of careful design and strategy work before building too many databases (ie do upfront architecture).
But then one of the group usually reveals that their company also did this and their architects are still trying to convince people that it is a good idea to build $100 million in infrastructure before seeing any benefits – and meanwhile projects are deciding they can’t wait for the utopian dream, so building “tactical data marts”.
These tactical data marts generally duplicate both data and interfaces without decommissioning, consolidating or even fixing existing data and interfaces.
So we end up with what is called spagettification-driven development.
Even with my limited exposure the business intelligence and data warehouses, I can see that this is both the wrong thing to do and an easy trap to fall into. So a better approach would be to merge detailed architecture (upfront and ongoing) with a continuous delivery of value (new useful reports, simplified data and decommissioned redundant interfaces).
With that in mind I thought I would publish some of the links that participants have sent me to research this further:
An ebook on “Agile data warehousing ” that several participants recommended (it will cost about $10)
A RUP based approach published by Scott Ambler
Cutter consorium page on business intelligence
Let me know if you have other links, or better yet some experience in successfully working with the implementation of data warehouses (using Agile or not).
Posted by James King