bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

Spring nested @Transactional rollback only

Recently we get some odd xxx Thread got an uncaught exception in Nagios alters. and the corresponding exception in log is :

Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Transaction marked as rollbackOnly org.springframework.transaction.TransactionSystemException: Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Transaction marked as rollbackOnly
    at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:521)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
    at methodA()...

Turns out the reason is we have a nested method also marked @Transactional and some exception happened inside which causes Spring marked it as RollBackonly in the thread local TransactionStatus.

@Transactional
public void outterMethod(){

 try{
  A a = new A();
  somedao.persist(a)
  ...
  nestedMethodWithTransationalAndHadExceptionInsde();
  ...}
 catch{
  //HERE, transaction is already marked as rollback only!
  somedao.merge(a);
 }
}

So the possible solution is:

  • remove the @Transacional from the nested method if it does not really require transaction control. So even it has exception, it just bubbles up and does not affect transactional stuff.
  • if nested method does need transaction control, make it as REQUIRE_NEW for the propagation policy that way even if throws exception and marked as rollback only, the caller will not be affected.

One caveat is :

Only unchecked exceptions (that is, subclasses of java.lang.RuntimeException) are rollbacked by default. For the case, a checked exception is thrown, the transaction will be committed!

And one customization can be done very easily by just adding the parameter rollBackFor to the @Transactional attribute:

@Transactional(rollbackFor = Exception.class)

Some source code from spring transactional implementation:

1. the Transaction abstraction

public interface PlatformTransactionManager {
    TransactionStatus getTransaction(TransactionDefinition definition) throws TransactionException;
    void commit(TransactionStatus status) throws TransactionException;
    void rollback(TransactionStatus status) throws TransactionException;
}

2. The TransactionDefinition

public interface TransactionDefinition {
    // Propagations
    int PROPAGATION_REQUIRED = 0;
    int PROPAGATION_SUPPORTS = 1;
    int PROPAGATION_MANDATORY = 2;
    int PROPAGATION_REQUIRES_NEW = 3;
    int PROPAGATION_NOT_SUPPORTED = 4;
    int PROPAGATION_NEVER = 5;
    int PROPAGATION_NESTED = 6;
    // Isolations
    int ISOLATION_DEFAULT = -1;
    int ISOLATION_READ_UNCOMMITTED = Connection.TRANSACTION_READ_UNCOMMITTED;
    int ISOLATION_READ_COMMITTED = Connection.TRANSACTION_READ_COMMITTED;
    int ISOLATION_REPEATABLE_READ = Connection.TRANSACTION_REPEATABLE_READ;
    int ISOLATION_SERIALIZABLE = Connection.TRANSACTION_SERIALIZABLE;
    // timeout
    int TIMEOUT_DEFAULT = -1;

    // behaviors
    int getPropagationBehavior();
    int getIsolationLevel();
    int getTimeout();
    boolean isReadOnly();
    String getName();
}

3. the TransactionStatus

public interface TransactionStatus extends SavepointManager, Flushable {
    boolean isNewTransaction();
    boolean hasSavepoint();
    void setRollbackOnly();
    boolean isRollbackOnly();
    void flush();
    boolean isCompleted();
}

A Chinese article about the source code.

 

logback spring boot rolling 配置

日志的配置

Spring Boot支持各种日志工具, 最常用的是Logback. 我们可以对日志进行配置, 由于日志是在ApplicationContext创建之前初始化的, 所以对日志的配置不能通过在@Configuration配置类上使用@PropertySources注解加载进来. 可以使用系统变量或者外部配置application.properties来加载. 配置文件中可以指定这些属性:

  • logging.config=: 配置文件的位置, 比如:classpath:logback.xml(logback的配置文件)
  • logging.file=: 日志文件名, 如:myapp.log, 输出日志到当前目录的myapp.log文件
  • logging.path=: 日志文件位置, 如:/var/log, 输出日志到/var/log/spring.log文件
  • logging.level.*=: 日志等级, 如:logging.level.org.springframework=DEBUG
  • logging.pattern.console=: 输出到console的日志格式, 只有logback有效
  • logging.pattern.file=: 输出到文件的日志格式, 只有logback有效
  • logging.pattern.level=: 日志级别的格式, 默认是%5p. 只有logback有效
  • logging.exception-conversion-word=%wEx: log异常时使用哪个格式转换器(base.xml中定义了三个conversionRule)
  • logging.register-shutdown-hook=false # Register a shutdown hook for the logging system when it is initialized(没用过)

上面这些属性配置, 一般写在application.properties中, 这样会被加载到Spring Environment中, 为了方便其他地方使用, Spring Environment中的一些属性也被转换到了系统属性(System property)里, 下面是这些属性于系统属性的对应关系:

Spring Environment

System Property

logging.exception-conversion-word

LOG_EXCEPTION_CONVERSION_WORD

logging.file

LOG_FILE

logging.path

LOG_PATH

logging.pattern.console

CONSOLE_LOG_PATTERN

logging.pattern.file

FILE_LOG_PATTERN

logging.pattern.level

LOG_LEVEL_PATTERN

PID

PID

日志配置文件

logging.config属性用于指定日志配置文件的位置, 以logback为例.

  • 如果不指定该属性, logback本身会默认寻找classpath下的配置文件, 寻找顺序为: logback.groovy > logback-test.xml > logback.xml;
  • Spring Boot又加了俩默认的配置文件:logback-spring.groovy > logback-spring.xml, 这俩优先级低于上面的那三个. 推荐指定使用logback-spring.xml.
  • 不指定配置文件时, 寻找上面的配置文件, 制定了则加载指定的配置文件. 如:logging.config=classpath:logback-abc.xml, 则会加载classpath下的logback-abc.xml文件

使用groovy需要添加groovy的包依赖:

<dependency>

    <groupId>org.codehaus.groovy</groupId>

    <artifactId>groovy</artifactId>

    <version>2.4.7</version>

</dependency>

输出到日志文件

logging.filelogging.path这俩属性用于指定日志文件输出的位置. 默认情况下Spring Boot只会把日志输出到console, 添加了这两个属性(任意一个即可), 才会把日志输出到文件里.

  • 两个属性都不指定, 只输出到控制台, 不输出到文件
  • logging.file指定文件, 可以是相对路径, 可以是绝对路径.
  • logging.path指定目录, 若制定了目录, 则会输出日志到指定目录下的spring.log文件中
  • 两个同时指定, 以logging.file为准

spring-boot包里关于logback的配置file-appender.xml中定义了文件输出到${LOG_FILE}, 在同一包下的base.xml文件里有这么一句:<property name=”LOG_FILE” value=”${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}”/>. 稍微分析下就知道为什么以logging.file为主, 指定logging.path时会输出到该目录下的spring.log文件里了.

注意上面语句中多次嵌套使用了${key:-defaultVlaue}形式

日志级别

logging.level.*用于指定日志级别, 比如:

logging.level.root=WARN

logging.level.org.springframework.web=DEBUG

logging.level.org.hibernate=ERROR

注意: 该属性配置的日志级别优先级要高于日志配置文件(如logback.xml), 即日志配置文件中与该属性定义的日志级别不一致时, 以该属性定义的级别为准.

日志格式

  • ogging.pattern.console指定在控制台输出的日志格式;
  • ogging.pattern.file指定在文件输出的日志格式;
  • ogging.pattern.level指定日之级别(DEBUG, INFO, ERROR等)的格式, 默认为%5p;

这些属性不指定时, 默认的格式在spring-boot包中的DefaultLogbackConfiguration类里有定义, 在defaults.xml里也有定义

格式大致为:

2016-11-02 21:59:11.366 INFO 11969 — [ main] o.apache.catalina.core.StandardService : Starting service Tomcat

依次为: 时间 日志级别 PID — [线程名] 日志名 : 日志内容

如何写自己的日志配置文件

spring-boot包里有四个相关的xml文件:

  • console-appender.xml: 定义了控制台输出的日志格式
  • file-appender.xml: 定义了一个日志的文件输出格式(指定每个文件10M)
  • defaults.xml: 定义了一些日志级别
  • base.xml: 包含了上面3个文件, 并指定了root的输出级别和输出方式

我们的日志配置线上不需要输出到console, 日志文件的大小一般也不会是10M, 所以上面那几个文件, 我们可以参考.

比如我们可以这样定义logback.xml

<?xml version=”1.0″ encoding=”UTF-8″?>

<configuration>

    <!– 这里面定义了 CONSOLE_LOG_PATTERN, FILE_LOG_PATTERN 等日志格式, 还定义了一些日志级别 –>

    <include resource=“org/springframework/boot/logging/logback/defaults.xml”/>

    <!– 命令行输出, 一般线上不用 –>

    <appender name=“CONSOLE” class=“ch.qos.logback.core.ConsoleAppender”>

        <encoder  charset=“UTF-8”>

            <pattern>${CONSOLE_LOG_PATTERN}</pattern>

        </encoder>

    </appender>

    <property name=“LOG_FILE_NAME” value=“myLog”/> <!– 定义一个属性, 下面用 –>

    <!– 输出格式 appender –>

    <appender name=“FILE” class=“ch.qos.logback.core.rolling.RollingFileAppender”>

        <file>${catalina.base}/logs/${LOG_FILE_NAME}.log</file>  <!– 可自己定义 –>

        <encoder>

            <pattern>${FILE_LOG_PATTERN}</pattern> <!– 输出格式也可自己定义 –>

        </encoder>

        <rollingPolicy class=“ch.qos.logback.core.rolling.TimeBasedRollingPolicy”>

            <fileNamePattern>${catalina.base}/logs/${LOG_FILE_NAME}.%d{yyyy-MM-dd}.log</fileNamePattern>

        </rollingPolicy>

    </appender>

    <!– error 日志 appender –>

    <appender name=“ERROR_FILE” class=“ch.qos.logback.core.rolling.RollingFileAppender”>

        <file>${catalina.base}/logs/${LOG_FILE_NAME}_error.log</file>

        <filter class=“ch.qos.logback.classic.filter.ThresholdFilter”>

            <level>ERROR</level>

        </filter>

        <encoder  charset=“UTF-8”>

            <pattern>${FILE_LOG_PATTERN}</pattern>

        </encoder>

        <rollingPolicy class=“ch.qos.logback.core.rolling.TimeBasedRollingPolicy”>

            <fileNamePattern>${catalina.base}/logs/${LOG_FILE_NAME}_error.%d{yyyy-MM-dd}.log</fileNamePattern>

        </rollingPolicy>

    </appender>

    <!– 定义日志级别, 也可在应用配置中指定 –>

    <logger name =“com.example.project” level=“INFO” />

    <logger name=“org.springframework.web” level=“DEBUG”/>

    <root level=“ERROR”>

        <appender-ref ref=“CONSOLE” /> <!– 线上不需要输出到 CONSOLE –>

        <appender-ref ref=“FILE” />

        <appender-ref ref=“ERROR_FILE” />

    </root>

</configuration>

  • 上例中, 日志会输出到文件XXX.log, 错误日志单独输出到一个XXX_error.log文件, 日志文件并每天打包一次.
  • 上例中, 应用配置(application.properties)里用于指定日志文件名文件位置的属性(logging.filelogging.path)将不起作用, 因为例子里没有用到这些属性, 其他配置(比如日志级别)仍有作用.
  • 上例中的哪个${catalina.base}算是一个系统变量, 表示应用所在目录, 文件名(位置)完全可以自己指定, 也可参考spring-boot包里的使用方式.

FROM HERE

Long Text Wrapping in ie9/10 inside table td

Both IE 9 and 10 seem to have problem wrapping long text inside table td element, even we explicitly set the the word-wrap CSS property.

Turns out the solution is we also have a set with and table-layout css to the wrapping table.

/* Both of them are Necessary.*/
.long-text-table {  
  table-layout:fixed;
  width:100%;
}
.long-text-wrapper{
  word-wrap:break-word;
}

HTML:

<table class="long-text-table">
 <tr>
  <td>
   <div class="long-text-wrapper"> REALLY LONG TEXT NEED TO WRAP</div>
  </td>
 </tr>
</table>

Optimize JPA one to many with fetch

 N+1

We all know the famous N+1 problem in the ORM world where the generated sql is not that optimized that we have to get the child collections with N number of separate sql query which leads to serious performance problem if there are many levels of objects tree.

So we can fix the problem in different ways

Let’s say we have an Parent object with a Set of Child objects.

1. Use JPA criteria fetch call

We can explicitly tell the criteria to do fetch instead of having separate query:

    public List&lt;Parent&gt; findAllParent()
    {
        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery&lt;Parent&gt; cq = cb.createQuery(Parent.class);
        Root&lt;Parent&gt; root = cq.from(Parent.class);
        root.fetch(Parent_.Child, JoinType.INNER);
        cq.distinct(true);
        return entityManager.createQuery(cq).getResultList();
    }

Note the distinct(true) is important here otherwise you get a lot of duplicate rows. reference HERE.

2. Use @Fetch(SUBSELECT or JOIN)

JPA doesn’t provide any specification on mapping annotations to select fetch strategy. In general, related entities can be fetched in any one of the ways given below

  • SELECT => one query for root entities + one query for related mapped entity/collection of each root entity = (n+1) queries, this is the DEFAULT
  • SUBSELECT => one query for root entities + second query for related mapped entity/collection of all root entities retrieved in first query = 2 queries
  • JOIN => one query to fetch both root entities and all of their mapped entity/collection = 1 query

So SELECT and JOIN are two extremes and SUBSELECT falls in between. One can choose suitable strategy based on her/his domain model.

By default SELECT is used by both JPA/EclipseLink and Hibernate. This can be overridden by using

@Fetch(FetchMode.JOIN)
@Fetch(FetchMode.SUBSELECT)

3. User @Fetch(SELECT) + @BatchSize

in Hibernate. It also allows to set SELECT mode explicitly using @Fetch(FetchMode.SELECT)which can be tuned by using batch size e.g. @BatchSize(size=10)

Corresponding annotations in EclipseLink are

@JoinFetch
@BatchFetch

stateful firewall with inbound outbound traffic

Background

I have worked as Devops for cloud migration in the recent 3 months without really writing much code. Even though being exposed to many AWS services like EMR/EC2/ASG(auto scaling group)/LC(launch config)/CF(cloud formation) etc.. with the need of setting up security groups(SG), i find myself still a bit confusing with inbound and outbound traffic rules. Was wondering if i allow inbound traffic, i have to send response back to client which means i have to allow outbound traffic? Did some google search with the question and get the keyword stateful firewall.

So basically with a stateful firewall, when a connection is established, the firewall will automatically let packets out back to the client’s port. You don’t need to create rules for that because the firewall knows.

History

Before the development of stateful firewalls, firewalls were stateless. A stateless firewall treats each network frame or packet individually. Such packet filters operate at the OSI Network Layer (layer 3) and function more efficiently because they only look at the header part of a packet.They do not keep track of the packet context such as the nature of the traffic. Such a firewall has no way of knowing if any given packet is part of an existing connection, is trying to establish a new connection, or is just a rogue packet. Modern firewalls are connection-aware (or state-aware), offering network administrators finer-grained control of network traffic.

Early attempts at producing firewalls operated at the application layer, which is the very top of the seven-layer OSI model. This method required exorbitant amounts of computing power and is not commonly used in modern implementations.

Description

A stateful firewall keeps track of the state of network connections (such as TCP streams or UDP communication) and is able to hold significant attributes of each connection in memory. These attributes are collectively known as the state of the connection, and may include such details as the IP addresses and ports involved in the connection and the sequence numbers of the packets traversing the connection. Stateful inspection monitors incoming and outgoing packets over time, as well as the state of the connection, and stores the data in dynamic state tables. This cumulative data is evaluated, so that filtering decisions would not only be based on administrator-defined rules, but also on context that has been built by previous connections as well as previous packets belonging to the same connection.

The most CPU intensive checking is performed at the time of setup of the connection. Entries are created only for TCP connections or UDP streams that satisfy a defined security policy. After that, all packets (for that session) are processed rapidly because it is simple and fast to determine whether it belongs to an existing, pre-screened session. Packets associated with these sessions are permitted to pass through the firewall. Sessions that do not match any policy are denied, as packets that do not match an existing table entry.

In order to prevent the state table from filling up, sessions will time out if no traffic has passed for a certain period. These stale connections are removed from the state table. Many applications therefore send keepalive messages periodically in order to stop a firewall from dropping the connection during periods of no user-activity, though some firewalls can be instructed to send these messages for applications.

Depending on the connection protocol, maintaining a connection’s state is more or less complex for the firewall. For example, TCP is inherently a stateful protocol as connections are established with a three-way handshake (“SYN, SYN-ACK, ACK”) and ended with a “FIN, ACK” exchange. This means that all packets with “SYN” in their header received by the firewall are interpreted to open new connections. If the service requested by the client is available on the server, it will respond with a “SYN-ACK” packet which the firewall will also track. Once the firewall receives the client’s “ACK” response, it transfers the connection to the “ESTABLISHED” state as the connection has been authenticated bidirectionally. This allows tracking of future packets through the established connection. Simultaneously, the firewall drops all packets which are not associated with an existing connection recorded in its state table (or “SYN” packets), preventing unsolicited connections with the protected machine by black hat hacking.

By keeping track of the connection state, stateful firewalls provide added efficiency in terms of packet inspection. This is because for existing connections the firewall need only check the state table, instead of checking the packet against the firewall’s rule set, which can be extensive. Additionally, in the case of a match with the state table, the firewall does not need to perform deep packet inspection.