Comments on: Should ETL really be ELT?

by: Mio

Tue, 29 Apr 2008 11:49:05 +0000

I would say TETLT is exactly what I’m doing in some of my cases. Not just the idea, proven practice in my case.

by: Rotimi Rufus

Wed, 10 Jan 2007 15:07:16 +0000

This may be a stale subject but I feel it necessary to add a word or two. I can’t agree less with Ian Bennett about the types of rules that can be stored in the database and shared via metadata with other applications. I have done ETL on a grand scale with OWB, SSIS and Data Integrator and one common denominator is that you cannot store or port complex rules without being platform (or database) - centric. ELT is good for what Andy described it for - keeping the business rules and relationships from the sources and being able to do a ‘look-back’ when necessary. ELT may not work well where it is not possible to connect the databases or have to use files to do the data transfers. You will also need to manage the refresh logic in from the source to the load area anyway.

by: Ian Bennett

Thu, 18 May 2006 23:37:00 +0000

Have to agree with Vincent. If transparency is important then what better than an ETL tool that is consistant regardless of the source or target? If the rules are stored in the database then you need to look in every database (not to mention understand the language used by different vendors) to get the full picture.

Also, all that an ETL tool need do is put more database specific features into their interfaces to get the performance benefits of ELT but the key point that many fail to see is that in most non-operational uses, efficiency/performance is not as much a priority as consistency, visiblility and scalability. ETL vendors are now introducing real-time functionality which is a farce when marketed for data warehousing but is important for positioning the ETL tools as middleware for all data movement which would mean more central control, visibility and flexibility and can support the MDM model for operational uses.

As far as the proprietary nature of ETL tools, I feel it is a a myth that in general the processes applied can be properly understood outside of the graphical presentation that these tools provide. Business rules in this domain are usually far more complex than A + B = C.

Personally I see ELT as a backwards step (hey, that’s what we used to do before ETL) unless they wrap it in a more consistent user interface in which case it just becomes a poor cousin to ETL tools because it doesn’t do anything that isnt possible in ETL yet does not have the flexibility provided by server processing.

by: Andy Hayler

Fri, 17 Mar 2006 12:26:00 +0000

Thanks also for your comment Nigel. Your idea of the TETLT made me smile; as you say, there will be some cases where the best order will vary.

By the way, whatever happened to Constellar? As I recall it had an excellent reputation, and was that rare beast: a UK enterprise software company. I know that they were bought in the end by DataMirror but wht was the inside story?

by: Andy Hayler

Fri, 17 Mar 2006 09:39:00 +0000

Some debate - excellent! The point you make about accessibility of the rules has some validity, but it varies by which ETL tool. Certainly in our client base we frequently encounter situations where complex ETL rules are regarded by customers as difficult to access and causing a major blockage to change. It is true that it MAY not be any better to store the business logic in the warehouse, but I believe that if properly implemented, this should be a more transparent mechanism. Moreover, there were two other reasons why ELT is likely to be superior to ETL most of the time: efficiency of using the DBMS engine, and (more importantly) retaining the different business structures, allowing the possibility at least of more flexible analysis.

by: Nigel Thomas (ex Oracle, ex Constellar Hub)

Fri, 17 Mar 2006 08:11:00 +0000

ETL has always been a misnomer anyway. You often need an element of T at the source; and yes, you often find that putting some T after the L helps, etc.

Which is why for a decade mostserious “ETL” tools have allowed you to specify schedules, chains or networks of these different elements (E, L, and all sorts of different Ts), often in a graphic way. Bolting an ETL/ELT hub onto a datawarehouse works in a lot of cases - but isn’t always the most appropriate solution.

So perhaps we should be talking about TETLT instead…?

by: Vincent McBurney

Thu, 16 Mar 2006 21:54:00 +0000

I’ve enjoyed following your blogs but I’ve taken exception to some of your comments this week. The ETL v ELT debate certainly has a lot of mileage left in it.

“transform rules that become difficult to maintain, in a set of metadata that may be hard to share with other applications”

Why are ELT transformation rules easier to maintain then ETL? Last time I checked transformation rules built into RDBMS views, procedures and functions were signficantly harder to document and maintain then ETL jobs, which is why ETL tools became popular in the first place.

What RDBMS or ELT tools have better metadata sharing and reporting then Informatica Superglue or WebSphere MetaStage?

regards
Vincent (ETL blogger :))