Comments on: ETL Method – Fastest Way To Get Data from DB2 to Microsoft SQL Server

By: Sue B

Sue B — Wed, 10 Feb 2010 13:14:19 +0000

I have changed a file transfer from ODBC that ran over 10 hours to rxferpcb that runs in 24 minutes. Thanks Steve!

By: hoospie

hoospie — Fri, 11 Dec 2009 11:47:31 +0000

This is a fine article. I got it all to work, but I haven't tried the COM object part. ave you done this? Did it create a strong performance increase?

By: Steve Novoselac

Steve Novoselac — Fri, 26 Jun 2009 11:50:38 +0000

you can tweak the settings in the export. Also you can just adjust your FMT file, or your columns in your destination, if you want to use numeric(8,0) or something

By: Mike

Mike — Thu, 25 Jun 2009 16:49:52 +0000

The output file contains fixed position columns with leading blanks on numeric fields. SQL Server format files do not provide for skipping these extra blanks. How do you handle this?

By: Steve Novoselac

Steve Novoselac — Wed, 24 Jun 2009 15:20:32 +0000

@Bryan – I am using an update date field. The tables I pull have a date field. The ETL I use only has to handle updates/inserts, no deletes. But I pull every greater than MAX update date in my SQL table subtract 3 days, and pull just the PK columns into a table, then do a delete by joining the PK’s to the main table in SQL Server. I have found that faster than looping through each row and doing a lookup/delete. Then I just grab all columns I want from db2 using the same logic, the max update date – 3, and insert into my table in SQL. Doing it that way, I get updates/inserts handled. You could use some kind of checksum method as well, I have blogged earlier about that, but for populating dimensions. There are other ways as well, using EXCEPT in SQL 2005, etc.

By: Bryan

Bryan — Tue, 23 Jun 2009 23:07:30 +0000

I’m having some issues wrapping my head around how you handled the incremental import. Looks like you are comparing an update field which you tagged during the previous load to a date in the DB2 database. I’ve been looking through the AS400 tables which I import for a field to compare to but cannot for the life of me find one.

I’d like to run the incrementals every 10 minutes so not sure the best way to approach this. Any insight would be greatly appreciated. Thanks!

By: Steve Novoselac

Steve Novoselac — Thu, 18 Jun 2009 03:55:25 +0000

I cant share it exactly, that is why I detailed it in the blog post. Your environment might be set up different than mine as well, so its best to follow the post and recreate it from that.

By: james

james — Thu, 18 Jun 2009 03:53:15 +0000

Hello Could you please share me this SSIS for me ?
I want to test this solution.If you can ,please let me know .

Thanks.

By: Duane Douglas

Duane Douglas — Fri, 22 May 2009 19:46:00 +0000

Thanks for the feedback!

I have another question: Can the .FMT file be used in a flat file source component as is, or does it first need some modification?

By: Steve Novoselac

Steve Novoselac — Fri, 22 May 2009 11:59:05 +0000

@Duane – sure. When I made my packages generic (allowed them to work for any table) I created a table called TESTFILE and set it up that way, just so it works. On a normal run, I pass in the actual table name from DB2 as a variable, and it changes the names on the tasks (replaces TESTFILE with the variable) so if I log execution I can see what is doing what. On the full load package, there are 3 steps, extract, truncate, load. The truncate is truncating that table in SQL server before I do a bulk load.