large file from hive to rdbms(oracle)

Recently we have a requirement of dumping a sizable file(4+G) to oracle from s3. The file itself is hive-compatiable. so instead of downloading the file and generate sql for it, we decided to transfer the content using hive jdbc and persist in via jpa/hiberante. Hive On the hive side, one important thing is to make…

batch / bulk insert/update in jpa/hibernate with flush and clear

For JPA use entity manager to do flush/clear For hibernate basically just switch the entity manager with hibernate session. When making new objects persistent flush() and then clear() the session regularly in order to control the size of the first-level cache. The suggest batch size is 20-50 by hibernate. However I found 1500 is good…