logback spring boot rolling 配置

日志的配置

Spring Boot支持各种日志工具, 最常用的是Logback. 我们可以对日志进行配置, 由于日志是在ApplicationContext创建之前初始化的, 所以对日志的配置不能通过在@Configuration配置类上使用@PropertySources注解加载进来. 可以使用系统变量或者外部配置application.properties来加载. 配置文件中可以指定这些属性:

  • logging.config=: 配置文件的位置, 比如:classpath:logback.xml(logback的配置文件)
  • logging.file=: 日志文件名, 如:myapp.log, 输出日志到当前目录的myapp.log文件
  • logging.path=: 日志文件位置, 如:/var/log, 输出日志到/var/log/spring.log文件
  • logging.level.*=: 日志等级, 如:logging.level.org.springframework=DEBUG
  • logging.pattern.console=: 输出到console的日志格式, 只有logback有效
  • logging.pattern.file=: 输出到文件的日志格式, 只有logback有效
  • logging.pattern.level=: 日志级别的格式, 默认是%5p. 只有logback有效
  • logging.exception-conversion-word=%wEx: log异常时使用哪个格式转换器(base.xml中定义了三个conversionRule)
  • logging.register-shutdown-hook=false # Register a shutdown hook for the logging system when it is initialized(没用过)

上面这些属性配置, 一般写在application.properties中, 这样会被加载到Spring Environment中, 为了方便其他地方使用, Spring Environment中的一些属性也被转换到了系统属性(System property)里, 下面是这些属性于系统属性的对应关系:

Spring Environment

System Property

logging.exception-conversion-word

LOG_EXCEPTION_CONVERSION_WORD

logging.file

LOG_FILE

logging.path

LOG_PATH

logging.pattern.console

CONSOLE_LOG_PATTERN

logging.pattern.file

FILE_LOG_PATTERN

logging.pattern.level

LOG_LEVEL_PATTERN

PID

PID

日志配置文件

logging.config属性用于指定日志配置文件的位置, 以logback为例.

  • 如果不指定该属性, logback本身会默认寻找classpath下的配置文件, 寻找顺序为: logback.groovy > logback-test.xml > logback.xml;
  • Spring Boot又加了俩默认的配置文件:logback-spring.groovy > logback-spring.xml, 这俩优先级低于上面的那三个. 推荐指定使用logback-spring.xml.
  • 不指定配置文件时, 寻找上面的配置文件, 制定了则加载指定的配置文件. 如:logging.config=classpath:logback-abc.xml, 则会加载classpath下的logback-abc.xml文件

使用groovy需要添加groovy的包依赖:

<dependency>

    <groupId>org.codehaus.groovy</groupId>

    <artifactId>groovy</artifactId>

    <version>2.4.7</version>

</dependency>

输出到日志文件

logging.filelogging.path这俩属性用于指定日志文件输出的位置. 默认情况下Spring Boot只会把日志输出到console, 添加了这两个属性(任意一个即可), 才会把日志输出到文件里.

  • 两个属性都不指定, 只输出到控制台, 不输出到文件
  • logging.file指定文件, 可以是相对路径, 可以是绝对路径.
  • logging.path指定目录, 若制定了目录, 则会输出日志到指定目录下的spring.log文件中
  • 两个同时指定, 以logging.file为准

spring-boot包里关于logback的配置file-appender.xml中定义了文件输出到${LOG_FILE}, 在同一包下的base.xml文件里有这么一句:<property name=”LOG_FILE” value=”${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}”/>. 稍微分析下就知道为什么以logging.file为主, 指定logging.path时会输出到该目录下的spring.log文件里了.

注意上面语句中多次嵌套使用了${key:-defaultVlaue}形式

日志级别

logging.level.*用于指定日志级别, 比如:

logging.level.root=WARN

logging.level.org.springframework.web=DEBUG

logging.level.org.hibernate=ERROR

注意: 该属性配置的日志级别优先级要高于日志配置文件(如logback.xml), 即日志配置文件中与该属性定义的日志级别不一致时, 以该属性定义的级别为准.

日志格式

  • ogging.pattern.console指定在控制台输出的日志格式;
  • ogging.pattern.file指定在文件输出的日志格式;
  • ogging.pattern.level指定日之级别(DEBUG, INFO, ERROR等)的格式, 默认为%5p;

这些属性不指定时, 默认的格式在spring-boot包中的DefaultLogbackConfiguration类里有定义, 在defaults.xml里也有定义

格式大致为:

2016-11-02 21:59:11.366 INFO 11969 — [ main] o.apache.catalina.core.StandardService : Starting service Tomcat

依次为: 时间 日志级别 PID — [线程名] 日志名 : 日志内容

如何写自己的日志配置文件

spring-boot包里有四个相关的xml文件:

  • console-appender.xml: 定义了控制台输出的日志格式
  • file-appender.xml: 定义了一个日志的文件输出格式(指定每个文件10M)
  • defaults.xml: 定义了一些日志级别
  • base.xml: 包含了上面3个文件, 并指定了root的输出级别和输出方式

我们的日志配置线上不需要输出到console, 日志文件的大小一般也不会是10M, 所以上面那几个文件, 我们可以参考.

比如我们可以这样定义logback.xml

<?xml version=”1.0″ encoding=”UTF-8″?>

<configuration>

    <!– 这里面定义了 CONSOLE_LOG_PATTERN, FILE_LOG_PATTERN 等日志格式, 还定义了一些日志级别 –>

    <include resource=“org/springframework/boot/logging/logback/defaults.xml”/>

    <!– 命令行输出, 一般线上不用 –>

    <appender name=“CONSOLE” class=“ch.qos.logback.core.ConsoleAppender”>

        <encoder  charset=“UTF-8”>

            <pattern>${CONSOLE_LOG_PATTERN}</pattern>

        </encoder>

    </appender>

    <property name=“LOG_FILE_NAME” value=“myLog”/> <!– 定义一个属性, 下面用 –>

    <!– 输出格式 appender –>

    <appender name=“FILE” class=“ch.qos.logback.core.rolling.RollingFileAppender”>

        <file>${catalina.base}/logs/${LOG_FILE_NAME}.log</file>  <!– 可自己定义 –>

        <encoder>

            <pattern>${FILE_LOG_PATTERN}</pattern> <!– 输出格式也可自己定义 –>

        </encoder>

        <rollingPolicy class=“ch.qos.logback.core.rolling.TimeBasedRollingPolicy”>

            <fileNamePattern>${catalina.base}/logs/${LOG_FILE_NAME}.%d{yyyy-MM-dd}.log</fileNamePattern>

        </rollingPolicy>

    </appender>

    <!– error 日志 appender –>

    <appender name=“ERROR_FILE” class=“ch.qos.logback.core.rolling.RollingFileAppender”>

        <file>${catalina.base}/logs/${LOG_FILE_NAME}_error.log</file>

        <filter class=“ch.qos.logback.classic.filter.ThresholdFilter”>

            <level>ERROR</level>

        </filter>

        <encoder  charset=“UTF-8”>

            <pattern>${FILE_LOG_PATTERN}</pattern>

        </encoder>

        <rollingPolicy class=“ch.qos.logback.core.rolling.TimeBasedRollingPolicy”>

            <fileNamePattern>${catalina.base}/logs/${LOG_FILE_NAME}_error.%d{yyyy-MM-dd}.log</fileNamePattern>

        </rollingPolicy>

    </appender>

    <!– 定义日志级别, 也可在应用配置中指定 –>

    <logger name =“com.example.project” level=“INFO” />

    <logger name=“org.springframework.web” level=“DEBUG”/>

    <root level=“ERROR”>

        <appender-ref ref=“CONSOLE” /> <!– 线上不需要输出到 CONSOLE –>

        <appender-ref ref=“FILE” />

        <appender-ref ref=“ERROR_FILE” />

    </root>

</configuration>

  • 上例中, 日志会输出到文件XXX.log, 错误日志单独输出到一个XXX_error.log文件, 日志文件并每天打包一次.
  • 上例中, 应用配置(application.properties)里用于指定日志文件名文件位置的属性(logging.filelogging.path)将不起作用, 因为例子里没有用到这些属性, 其他配置(比如日志级别)仍有作用.
  • 上例中的哪个${catalina.base}算是一个系统变量, 表示应用所在目录, 文件名(位置)完全可以自己指定, 也可参考spring-boot包里的使用方式.

FROM HERE

Long Text Wrapping in ie9/10 inside table td

Both IE 9 and 10 seem to have problem wrapping long text inside table td element, even we explicitly set the the word-wrap CSS property.

Turns out the solution is we also have a set with and table-layout css to the wrapping table.

/* Both of them are Necessary.*/
.long-text-table {  
  table-layout:fixed;
  width:100%;
}
.long-text-wrapper{
  word-wrap:break-word;
}

HTML:

<table class="long-text-table">
 <tr>
  <td>
   <div class="long-text-wrapper"> REALLY LONG TEXT NEED TO WRAP</div>
  </td>
 </tr>
</table>

Optimize JPA one to many with fetch

 N+1

We all know the famous N+1 problem in the ORM world where the generated sql is not that optimized that we have to get the child collections with N number of separate sql query which leads to serious performance problem if there are many levels of objects tree.

So we can fix the problem in different ways

Let’s say we have an Parent object with a Set of Child objects.

1. Use JPA criteria fetch call

We can explicitly tell the criteria to do fetch instead of having separate query:

    public List&lt;Parent&gt; findAllParent()
    {
        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery&lt;Parent&gt; cq = cb.createQuery(Parent.class);
        Root&lt;Parent&gt; root = cq.from(Parent.class);
        root.fetch(Parent_.Child, JoinType.INNER);
        cq.distinct(true);
        return entityManager.createQuery(cq).getResultList();
    }

Note the distinct(true) is important here otherwise you get a lot of duplicate rows. reference HERE.

2. Use @Fetch(SUBSELECT or JOIN)

JPA doesn’t provide any specification on mapping annotations to select fetch strategy. In general, related entities can be fetched in any one of the ways given below

  • SELECT => one query for root entities + one query for related mapped entity/collection of each root entity = (n+1) queries, this is the DEFAULT
  • SUBSELECT => one query for root entities + second query for related mapped entity/collection of all root entities retrieved in first query = 2 queries
  • JOIN => one query to fetch both root entities and all of their mapped entity/collection = 1 query

So SELECT and JOIN are two extremes and SUBSELECT falls in between. One can choose suitable strategy based on her/his domain model.

By default SELECT is used by both JPA/EclipseLink and Hibernate. This can be overridden by using

@Fetch(FetchMode.JOIN)
@Fetch(FetchMode.SUBSELECT)

3. User @Fetch(SELECT) + @BatchSize

in Hibernate. It also allows to set SELECT mode explicitly using @Fetch(FetchMode.SELECT)which can be tuned by using batch size e.g. @BatchSize(size=10)

Corresponding annotations in EclipseLink are

@JoinFetch
@BatchFetch

stateful firewall with inbound outbound traffic

Background

I have worked as Devops for cloud migration in the recent 3 months without really writing much code. Even though being exposed to many AWS services like EMR/EC2/ASG(auto scaling group)/LC(launch config)/CF(cloud formation) etc.. with the need of setting up security groups(SG), i find myself still a bit confusing with inbound and outbound traffic rules. Was wondering if i allow inbound traffic, i have to send response back to client which means i have to allow outbound traffic? Did some google search with the question and get the keyword stateful firewall.

So basically with a stateful firewall, when a connection is established, the firewall will automatically let packets out back to the client’s port. You don’t need to create rules for that because the firewall knows.

History

Before the development of stateful firewalls, firewalls were stateless. A stateless firewall treats each network frame or packet individually. Such packet filters operate at the OSI Network Layer (layer 3) and function more efficiently because they only look at the header part of a packet.They do not keep track of the packet context such as the nature of the traffic. Such a firewall has no way of knowing if any given packet is part of an existing connection, is trying to establish a new connection, or is just a rogue packet. Modern firewalls are connection-aware (or state-aware), offering network administrators finer-grained control of network traffic.

Early attempts at producing firewalls operated at the application layer, which is the very top of the seven-layer OSI model. This method required exorbitant amounts of computing power and is not commonly used in modern implementations.

Description

A stateful firewall keeps track of the state of network connections (such as TCP streams or UDP communication) and is able to hold significant attributes of each connection in memory. These attributes are collectively known as the state of the connection, and may include such details as the IP addresses and ports involved in the connection and the sequence numbers of the packets traversing the connection. Stateful inspection monitors incoming and outgoing packets over time, as well as the state of the connection, and stores the data in dynamic state tables. This cumulative data is evaluated, so that filtering decisions would not only be based on administrator-defined rules, but also on context that has been built by previous connections as well as previous packets belonging to the same connection.

The most CPU intensive checking is performed at the time of setup of the connection. Entries are created only for TCP connections or UDP streams that satisfy a defined security policy. After that, all packets (for that session) are processed rapidly because it is simple and fast to determine whether it belongs to an existing, pre-screened session. Packets associated with these sessions are permitted to pass through the firewall. Sessions that do not match any policy are denied, as packets that do not match an existing table entry.

In order to prevent the state table from filling up, sessions will time out if no traffic has passed for a certain period. These stale connections are removed from the state table. Many applications therefore send keepalive messages periodically in order to stop a firewall from dropping the connection during periods of no user-activity, though some firewalls can be instructed to send these messages for applications.

Depending on the connection protocol, maintaining a connection’s state is more or less complex for the firewall. For example, TCP is inherently a stateful protocol as connections are established with a three-way handshake (“SYN, SYN-ACK, ACK”) and ended with a “FIN, ACK” exchange. This means that all packets with “SYN” in their header received by the firewall are interpreted to open new connections. If the service requested by the client is available on the server, it will respond with a “SYN-ACK” packet which the firewall will also track. Once the firewall receives the client’s “ACK” response, it transfers the connection to the “ESTABLISHED” state as the connection has been authenticated bidirectionally. This allows tracking of future packets through the established connection. Simultaneously, the firewall drops all packets which are not associated with an existing connection recorded in its state table (or “SYN” packets), preventing unsolicited connections with the protected machine by black hat hacking.

By keeping track of the connection state, stateful firewalls provide added efficiency in terms of packet inspection. This is because for existing connections the firewall need only check the state table, instead of checking the packet against the firewall’s rule set, which can be extensive. Additionally, in the case of a match with the state table, the firewall does not need to perform deep packet inspection.

My Postgresql snippets

Install psql

To get the psql, I simply installed the postgresapp for OSX which bundles the psql . We can start the psql from its GUI, or added its bin to our PATH. The specific path is: /Applications/Postgres.app/Contents/Versions/latest/bin

You can also get psql by installing postgresql one click installer.

Connect to DB with psql

psql -h DbUrl -p DbPort -U DbUser

typically a password prompt comes up(if not add -W to force it).

skip password

If we need to avoid entering pw interactively, we could set the pw as PGPASSWORD in the PATH so that psql could pickup.

export PGPASSWORD='password'

For all other environment variables, reference here.

run batch sql file

append -a -f YourScript.sql to the above command.

-a will print all nonempty input lines to standard output as they are read.

After the file is processed, psql terminates.

Or in the psql shell, \i path_to_sql_file

quit psql shell

use \quite

meta sqls

All DBs: select datname from pg_database;

All Tables: select tablename from pg_table where tableowner='YOUROWNER';

describe table: select column_name, data_type, character_maximum_length from INFORMATION_SCHEMA.COLUMNS where table_name = 'YourTableName';
Or use  \d schemaName.tableName in psql shell.

install ora2pg on mac osx

Recently we need to migrate our on prem oracle database into postgresql on aws RDS. Looks like ora2pg is a nice little tool to mitigate the problem though not perfect.

I am running OS X 10.10.5 Yosemite. It already comes with perl v5.18.2 as well as cpan for installing packages.

download/install Oracle client

The official document specifies quite vaguely about what is the oracle client libraries.  Turns out we need to download 3 packages from oracle website, for my use case we have oracle 12c so my current version is 12.1.02. The packages you need are:

  1. Oracle Basic
  2. Sqlplus
  3. sdk

Then we unzip them to the SAME directory like: /Users/liha/develop/instantclient_12_1 .

Now we need to set 2 env variables for installation:

export LD_LIBRARY_PATH=/Users/liha/develop/instantclient_12_1
export ORACLE_HOME=/Users/liha/develop/instantclient_12_1

install DBI/DBD

Now we need to install DBI/DBD stuff that is required during the ora2pg Makefile phase, especially the DBD::Oracle. The DBD:Pg is optional, only if you need to dump data/schema live rather than offline to file.

#perl –MCPAN –e shell
…………
cpan> install DBI
…………
cpan> install DBD::Oracle
…………
cpan> install DBD::Pg
…………

Check module

Now we can check whether the dependencies are correctly installed by running a perl script.

 #!/usr/bin/perl
use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();

result should look like something:

➜  ora2pg perl test.pl                                                                                                                                                                              12:48:46
Bundle::NetSNMP
DBD::Oracle
DBI
Ora2Pg
Perl⏎

install ora2Pg

now can start to install the real thing by download and extract the ora2Pg tar.bz2 and run its makefile.pl which will eventually generate our makefile which we can make and install:

perl Makefile.PL
make && make install

After installation, the default conf file will be under etc/ora2pg/ora2pg.conf.

ora2Pg config

Now we can start config the ora2Pg by changing the default conf file or create a new conf file based on default conf. If you have multiple DB schemas to migrate or have multiple environments, it would be better to create different conf for each use case.

The important fields are:

ORACLE_DSN      dbi:Oracle:host=hostname;sid=Oracle SID

ORACLE_USER     schema_owner name

ORACLE_PWD      password

You should set as below to connect and retrieve required schema data and objects

USER_GRANTS     1

I also turned debug on to view progress

DEBUG           1

Other settings used were

EXPORT_SCHEMA   1

SCHEMA          schema_owner_name

CREATE_SCHEMA   1

I also used the same schema name in PG

PG_SCHEMA       schema_owner_name

I did have some problems with characters not being copied across as required and had to use settings below to override defaults, on some occasions

NLS_LANG       AMERICAN_AMERICA.UTF8

NLS_CHAR        LATIN1

Now you can run the dump:

ora2pg -c myOra2pg.conf -t TABLE -o tables.sql

另外一个中文文章