java volatile 可见行(Visibility),原子性(Atomicity),有序性(Ordering)

Saw a very good Chinese article explaining Visibility/Atomicity/Ordering for the volatile keyword.  From HERE. Summary is Volatile provides visibility and ordering but not atomicity. Locking guarantees atomicity, visibility and ordering. So volatile is not a substitute for synchronization. HERE is another article from one of the jsr-133 author written back in 2007 but still very informative. 一、Java内存模型 想要理解volatile为什么能确保可见性,就要先理解Java中的内存模型是什么样的。…

regex dash – inside bracket []

We have a requirement to remove some illegal character in file name like slash etc… So I put on a regex in our code: [^a-zA-Z0-9.-_] I assume the above regex will match everything that is not number/char/dot/dash/underscore. Turns out I was wrong! The – is special case inside […] that is used for range. It should be…

print all data in paginated table/grid

direct Tabular data display Recently our project has a page need to show tabular data from 30-6000 rows with 3-4 columns. At first, I thought this is pretty reasonable data to show in one page so I just throw the data into a ng-repeat table with my own implementation of filtering/sorting which is pretty straightforward in…

Understand optional true in maven dependency

Today one colleague  from the other team was trying to mimic our behavior doing custom authentication on Hive-Server2. He asked me why he could not get the HiveConf.get(key) working. It basically gets the key we defined in hive-site.xml. It is convenient because if we put key/value there, we do not have to worry about path issue…

Serverless EMR Cluster monitoring with Lambda

Background One issue we typically have is our EMR cluster stops consuming hive queries due to the overload of the metastore loading/refreshing. This is partially caused by the usage of the shared-metastore which hosts many teams’ schema/tables inside our organization. When this happens in prod, we have to ask help from RIM to terminate our…

large file from hive to rdbms(oracle)

Recently we have a requirement of dumping a sizable file(4+G) to oracle from s3. The file itself is hive-compatiable. so instead of downloading the file and generate sql for it, we decided to transfer the content using hive jdbc and persist in via jpa/hiberante. Hive On the hive side, one important thing is to make…

Spring nested @Transactional rollback only

Recently we get some odd xxx Thread got an uncaught exception in Nagios alters. and the corresponding exception in log is : Turns out the reason is we have a nested method also marked @Transactional and some exception happened inside which causes Spring marked it as RollBackonly in the thread local TransactionStatus. So the possible solution is:…

logback spring boot rolling 配置

日志的配置 Spring Boot支持各种日志工具, 最常用的是Logback. 我们可以对日志进行配置, 由于日志是在ApplicationContext创建之前初始化的, 所以对日志的配置不能通过在@Configuration配置类上使用@PropertySources注解加载进来. 可以使用系统变量或者外部配置application.properties来加载. 配置文件中可以指定这些属性: logging.config=: 配置文件的位置, 比如:classpath:logback.xml(logback的配置文件) logging.file=: 日志文件名, 如:myapp.log, 输出日志到当前目录的myapp.log文件 logging.path=: 日志文件位置, 如:/var/log, 输出日志到/var/log/spring.log文件 logging.level.*=: 日志等级, 如:logging.level.org.springframework=DEBUG logging.pattern.console=: 输出到console的日志格式, 只有logback有效 logging.pattern.file=: 输出到文件的日志格式, 只有logback有效 logging.pattern.level=: 日志级别的格式, 默认是%5p. 只有logback有效 logging.exception-conversion-word=%wEx: log异常时使用哪个格式转换器(base.xml中定义了三个conversionRule) logging.register-shutdown-hook=false # Register a shutdown hook for the logging system when it is initialized(没用过) 上面这些属性配置, 一般写在application.properties中, 这样会被加载到Spring Environment中, 为了方便其他地方使用, Spring…

Optimize JPA one to many with fetch

 N+1 We all know the famous N+1 problem in the ORM world where the generated sql is not that optimized that we have to get the child collections with N number of separate sql query which leads to serious performance problem if there are many levels of objects tree. So we can fix the problem…

serialize enum fields with gson

By default, Gson just serialize the ‘name’ of the Enum which might not be enough since we might need also want to carry all the fields during the serialization. To achieve this we need to has our own gson adaptor and make use of reflection. usage: Enum class: Test Code: Result:

JVM JIT and running mode

JIT At the beginning of Java, all code are executed in interpreted manner, I.E execute code line by line after interpretation which would result in slowness. Especially for code that are frequently executed. So later JIT was introduced so that when when some code are frequently executed, it becomes ‘Hot Spot Code’ and would be…