Comments on: SSIS – Slowly Changing Dimensions with Checksum http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ Homepage of Steve Novoselac Fri, 12 Mar 2010 15:59:49 -0800 http://wordpress.org/?v=2.9.2 hourly 1 By: Maintaining a Type 1 Slowly Changing Dimension (SCD) using T-SQL « laqua dot us http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/comment-page-1/#comment-160522 Maintaining a Type 1 Slowly Changing Dimension (SCD) using T-SQL « laqua dot us Tue, 24 Nov 2009 21:42:13 +0000 http://blog.stevienova.com/?p=1010#comment-160522 [...] of the SCD wizard to speed things up, maybe even some fancy checksum voodoo for the updates (see http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ for an example). Then after thinking about it a little more – why are we sending a million [...] [...] of the SCD wizard to speed things up, maybe even some fancy checksum voodoo for the updates (see http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ for an example). Then after thinking about it a little more – why are we sending a million [...]

]]> By: Todd McDermid http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/comment-page-1/#comment-160485 Todd McDermid Mon, 28 Sep 2009 20:58:13 +0000 http://blog.stevienova.com/?p=1010#comment-160485 You may also want to try out a single-component SCD: the Kimball Method SCD (<a href="http://kimballscd.codeplex.com" rel="nofollow">http://kimballscd.codeplex.com</a>). I've experienced a hundred-fold increase in performance, and I haven't added checksumming to it yet, although other users have, and report even higher performance gains... You may also want to try out a single-component SCD: the Kimball Method SCD (http://kimballscd.codeplex.com). I've experienced a hundred-fold increase in performance, and I haven't added checksumming to it yet, although other users have, and report even higher performance gains…

]]> By: Maintaining a Type 1 Slowly Changing Dimension (SCD) using T-SQL | tim laqua dot com http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/comment-page-1/#comment-156464 Maintaining a Type 1 Slowly Changing Dimension (SCD) using T-SQL | tim laqua dot com Sat, 23 May 2009 18:16:08 +0000 http://blog.stevienova.com/?p=1010#comment-156464 [...] of the SCD wizard to speed things up, maybe even some fancy checksum voodoo for the updates (see http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ for an example). Then after thinking about it a little more - why are we sending a million rows [...] [...] of the SCD wizard to speed things up, maybe even some fancy checksum voodoo for the updates (see http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ for an example). Then after thinking about it a little more – why are we sending a million rows [...]

]]> By: John Shanks http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/comment-page-1/#comment-143104 John Shanks Thu, 26 Feb 2009 22:25:00 +0000 http://blog.stevienova.com/?p=1010#comment-143104 I would have to agree with you Steve that the SCD functionality, in SSIS is slow. I saw even worse performance than you described but switched to using the Checksum, Lookup, Conditional Split approach and the performance improved dramatically. I saw 200 thousand rows go from 10 minutes to 2 minutes. I would have to agree with you Steve that the SCD functionality, in SSIS is slow. I saw even worse performance than you described but switched to using the Checksum, Lookup, Conditional Split approach and the performance improved dramatically. I saw 200 thousand rows go from 10 minutes to 2 minutes.

]]> By: Marco http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/comment-page-1/#comment-143094 Marco Thu, 26 Feb 2009 21:46:57 +0000 http://blog.stevienova.com/?p=1010#comment-143094 Very nice write up Steve. I just recently dove into SSIS and SCD ... I must say I'm very impressed w/the technology as a whole. A couple points I wanted to relay, please provide feedback when convenient: 1) Regarding your performance figures: I've resorted to using the SCD Wizard. We're getting completion times of around 10 - 20 minutes for target dimension tables that are 300+ million rows, now closely approaching 400 million rows. Each write (the SCD packages run every night @ 1 am) handles updating about 8k - 12k rows. This is on a 8 cpu/32 gig machine. What hardware profile have you based your performance figures on? 2) Regarding the lookup transformation, you've explicitly created this in your solution...and I see you're explicitly handling new rows that don't exist in your destination table. Is a similar transformation created "behind the scenes" when using the SCD wizard (I didn't see the Lookup Transformation operation box in any of my SCD packages). One of my SCD packages has recently started failing, citing the error "The Lookup transformation failed to retrieve any rows." It looks like existing records are updated fine ... so do I need to handle new records that don't exist in the destination table explicitly? ... do you have any experience w/treating such an error? Very nice write up Steve. I just recently dove into SSIS and SCD … I must say I’m very impressed w/the technology as a whole. A couple points I wanted to relay, please provide feedback when convenient:

1) Regarding your performance figures: I’ve resorted to using the SCD Wizard. We’re getting completion times of around 10 – 20 minutes for target dimension tables that are 300+ million rows, now closely approaching 400 million rows. Each write (the SCD packages run every night @ 1 am) handles updating about 8k – 12k rows. This is on a 8 cpu/32 gig machine. What hardware profile have you based your performance figures on?

2) Regarding the lookup transformation, you’ve explicitly created this in your solution…and I see you’re explicitly handling new rows that don’t exist in your destination table. Is a similar transformation created “behind the scenes” when using the SCD wizard (I didn’t see the Lookup Transformation operation box in any of my SCD packages). One of my SCD packages has recently started failing, citing the error “The Lookup transformation failed to retrieve any rows.” It looks like existing records are updated fine … so do I need to handle new records that don’t exist in the destination table explicitly? … do you have any experience w/treating such an error?

]]>