Understand optional true in maven dependency

Today one colleague  from the other team was trying to mimic our behavior doing custom authentication on Hive-Server2. He asked me why he could not get the HiveConf.get(key) working. It basically gets the key we defined in hive-site.xml. It is convenient because if we put key/value there, we do not have to worry about path issue in cluster, just do HiveConf c = new HiveConf() and call get(key) . (side note: this way seems to be not recommended officially since now they have a bunch of enum value to restrict what you can define there) After looking at the source code. get() is actually a method in the parent Configuration class.

import org.apache.hadoop.conf.Configuration;
public class HiveConf extends Configuration

On my pom, I explicitly included the

       <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-core</artifactId>
         <version>${hadoop.core.version}</version>
      </dependency>

So i can look at it from IDE directly. However there is no such dependency in his pom, I cannot find the artifact from the dependency tree either. This is really confusing to me.

It turns out the org.apache.hive -> hive-service has dependency on some hive-shims artifacts which has hadoop-core dependency specified as optional=true.

   <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-core</artifactId>
      <version>${hadoop-20.version}</version>
<optional>true</optional>
    </dependency>

so what is optional true? How come his jar can compile even without the hadoop-core dependency specified? The below pictures are from this POST.

Meaning of <optional>

In short, if project D depend on project C, Project C optionally depend on project A, then project D do NOT depend on project A.

image

Since project C has 2 classes use some classes from project A and project B. Project C can not get compiled without dependencies on A and B. But these two classes are only optional features, which may not be used at all in project D, which depend on project C. So to make the final war/ejb package don’t contain unnecessary dependencies, use to indicate the dependency is optional, be default will not be inherited by others.

What happens if project D really used OptionaFeatureOne in project C? Then in project D‘s pom, project A need to be explicitly declared in the dependencies section.

image

If optional feature one is used in project D, then project D‘s pom need to declare dependency on project A to pass compile. Also, the final war package of project D doesn’t contain any class from project B, since feature 2 is now used.

Our case

In our scenario, the hive-service depends on hive-exec which depends on hive-shims which has optional dependency on hadoop-core. So when the Configuration.get() is not used, his project could still compile even though HiveConf extends the Configuration class. Now if the get() method is to be used, then he need to explicitly declare the dependency on hadoop-core.

Advertisements

exclude xml-apis dependency in maven

The xml-apis version 1.0xxx is referenced by multiple hibernate artifacts like hibernate-core, hibernate-entitymanager etc. It is annoying because it would conflict with the JRE’s own javax.xml api in the rt.jar which would cause problems like xml cannot parse.

We need to be extra careful about it since it usually is introduced thru transitive dependencies. Definitely manual exclude it from the known artifacts would help, but we can have a maven plugin to check for us: maven-enforcer-plugin

If we have similar config as below and have the xml-apis.jar in the dependency tree, the build will fail.

         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-enforcer-plugin</artifactId>
            <version>1.4.1</version>
            <executions>
               <execution>
                  <id>enforce-banned-dependencies</id>
                  <goals>
                     <goal>enforce</goal>
                  </goals>
                  <configuration>
                     <rules>
                        <bannedDependencies>
                           <excludes>
                              <!--this is to check we do not have the xml-apis included since JRE provides it already-->
                              <exclude>xml-apis:xml-apis</exclude>
                           </excludes>
                        </bannedDependencies>
                     </rules>
                     <fail>true</fail>
                  </configuration>
               </execution>
            </executions>
         </plugin>

Some other interesting discussion on this about maven’s nearest win policy.

version compress css js with maven

One problem in release is the user still caches our resources that our new look/feel can not be presented unless user hard refresh the browser.

To overcome this, several ways:

1. append a query string(like UUID) to the resource

This is good if you want to user download a fresh copy each request. If we randomize the query string with every request, this will defeat caching entirely. This is rarely/never desirable for a production application’s CSS.

Add a <%@ page import=”java.util.UUID” %> to the head of jsp and append the uuid to the resource by adding:

<script type=”text/javascript” language=”javascript” src=”dir/gwt_whatever.nocache.js?<%=UUID.randomUUID().toString()%>“></script>

2. use a new name each build

The yuicompressor-maven-plugin has a <suffix> option to let you add suffix to the resource after compression.

3.append a query String (like build date) to the resource

This is actually the way i used. The good part is it still keeps the file name, user could still get a new version for the each release. To achieve this.

3.1 add a property in the pom. This way we have a timestamp var available.

		<timestamp>${maven.build.timestamp}</timestamp>
		<maven.build.timestamp.format>yyyy-MM-dd</maven.build.timestamp.format>

3.2 add resource in the war plugin to filter the web.xml so that we could replace variables

<webResources>
    <!--compressed resources, see yuicompressor-maven-plugin below for detail-->
    <resource>
        <directory>${project.build.directory}/min</directory>
    </resource>
    <!--this is to replace the ${...} with our maven properties. For example the timestamp property replacement in the web.xml-->
    <resource>
        <directory>${basedir}/src/main/webapp/WEB-INF</directory>
        <filtering>true</filtering>
        <targetPath>WEB-INF</targetPath>
        <includes>
            <include>**/web.xml</include>
        </includes>
    </resource>
</webResources>

For the yuicompress maven plugin config:

			
<plugin>
    <groupId>net.alchim31.maven</groupId>
    <artifactId>yuicompressor-maven-plugin</artifactId>
    <version>1.5.1</version>
    <executions>
        <execution>
            <goals>
                <goal>compress</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <nosuffix>true</nosuffix>
        <excludes>
            <exclude>**/*.min.js</exclude>
            <exclude>**/jquery*.js</exclude>
            <exclude>**/angular*.js</exclude>
            <exclude>**/*.min.css</exclude>
            <exclude>**/jquery*.css</exclude>
        </excludes>
        <!--maven war plugin will overwrite the file we compressed, so we add a 'webappDirectory' here to put them into the /min directory and
					    then in the maven-war-plugin define this /min directory as a <resource> to make sure they could be included-->
        <webappDirectory>${project.build.directory}/min</webappDirectory>
    </configuration>
</plugin>

3.3 Add an init-param in the web.xml. Note the ${timestamp} will be replaced by the maven time stamp because of the filtering config we have in 3.2 and the property definition in 3.1

  <context-param>
    <param-name>buildTimeStamp</param-name>
    <param-value>${timestamp}</param-value>
  </context-param>

3.4 now the ‘buildTimeStamp’ is available for us in the jsp context, we just need to include it:

<link rel="stylesheet"
href="/css/whatever.css?build=<c:out value='${initParam.buildTimeStamp}'/>"
type="text/css">

 

maven build phase plugine(clover duplicate class)

When using clover2 in CI build, it might throw duplicate class error if using: mvn clover2:setup test clover2:aggregate clover2:clover.

The reason is the clover2:setup is by default in the generate-source phase will copy the src to target/clover/src-instrumented and the test will also run the generate source and compile. Now there end up to be 2 source and duplicate class error is thrown.

One solution is to bind the clover2:setup to the process-source phase.

<executions>
        <execution>
            <id>instrument</id>
            <phase>process-sources</phase>
            <goals>
                <goal>setup</goal>
            </goals>
        </execution>
    </executions>

More about maven phase and plugin

Maven helps you build a project. The way it does that is through the build lifecycle and theplugins.

The lifecycle is made of phases that you can call explicitly on the command line, for example:

mvn package

package is a phase part of the default build lifecycle, like compile ordeploy. All the phases of the default lifecycle can be found in thereference. At each phase, Maven calls a goal in a plugin that does something for you. For example, the maven-compiler-plugin has acompile goal that compiles your java code during the compile phase of the lifecycle. You can also explicitly call a plugin on the command line, like:

mvn clean:clean

which calls the clean goal on the maven-clean-plugin. By default this goal is bound to the clean phase, so you can call it by executing mvn clean. You can call any plugin by using its group id, its artifact id, its version and the goal you want to execute, e.g.:

mvn org.codehaus.mojo:versions-maven-plugin:2.1:set -DnewVersion=1.2.3

(A plugin named blah-maven-plugin can be called by the shortened version of its name, blah. See also the Guide to Developing Java Plugins)

To “bind” a plugin goal to a phase, you just need to define your plugin in your pom.xml, and define its execution to a phase of the lifecycle:

      <plugin>
        <groupId>org.mirah.maven</groupId>
        <artifactId>maven-mirah-plugin</artifactId>
        <version>1.1-SNAPSHOT</version>
        <executions>
          <execution>
            <phase>compile</phase>
            <goals><goal>compile</goal></goals>
          </execution>
        </executions>
      </plugin>

Here we are attaching the mirah compile goal to the compile phase of the lifecycle. When Maven executes the compile phase, it will the compile the Mirah code for us.

switch min.js from development to production

we usually need the full js for development purpose, but the min.js for production.

same thing applies to CSS.

The way we are using is thru maven-replacer-plugin

<!-- DEVELOPMENT -->
<script src="lib/lib1.js"></script>
<script src="lib/lib2.js"></script>
<script src="js/js1.js"></script>
<script src="js/js2.js"></script>
<!-- /DEVELOPMENT -->
<!-- PRODUCTION
<script src="lib/lib1.min.js"></script>
<script src="//cdn/lib2.min.js"></script>
<script src="js/combined.min.js"></script>
/PRODUCTION -->

replacer plugin config:

The phase here need to the prepare-package. If you use package, the build will pass BUT only the file in the exploded will be replaced, the file in the war will not.

       			         <!--replace plugin so that we could switch our JS from development to production min.js. we do this before maven package the exploded to
             the war(prepare-package) phase. REF-> https://code.google.com/p/maven-replacer-plugin/wiki/UsageWithOtherPlugins-->
         <plugin>
            <groupId>com.google.code.maven-replacer-plugin</groupId>
            <artifactId>replacer</artifactId>
            <version>${maven.replacer.version}</version>
            <executions>
               <execution>
                  <phase>prepare-package</phase>
                  <goals>
                     <goal>replace</goal>
                  </goals>
               </execution>
            </executions>
            <configuration>
               <file>${project.build.directory}/${project.build.finalName}/WEB-INF/jsp/inc/angular-bootstrap-inc.jsp</file>
               <replacements>
                  <replacement>
                     <token>&lt;%-- DEVELOPMENT --%&gt;</token>
                     <value>&lt;%-- DEVELOPMENT</value>
                  </replacement>                  <replacement>
                     <token>&lt;%-- /DEVELOPMENT --%&gt;</token>
                     <value> /DEVELOPMENT --%&gt;</value>
                  </replacement>
                  <replacement>
                     <token>&lt;%-- PRODUCTION</token>
                     <value>&lt;%-- PRODUCTION --%&gt;</value>
                  </replacement>                  <replacement>
                     <token>/PRODUCTION --%&gt;</token>
                     <value>&lt;%-- /PRODUCTION --%&gt;</value>
                  </replacement>
               </replacements>
            </configuration>
         </plugin>

After the prod profile build:

<!-- DEVELOPMENT
<script src="lib/lib1.js"></script>
<script src="lib/lib2.js"></script>
<script src="js/js1.js"></script>
<script src="js/js2.js"></script>
/DEVELOPMENT -->
<!-- PRODUCTION -->
<script src="lib/lib1.min.js"></script>
<script src="//cdn/lib2.min.js"></script>
<script src="js/combined.min.js"></script>
<!-- /PRODUCTION -->

Maven war plugin config

I want to keep the local copy as development mode, so i basically replace the development with production when using the normal profile.

note the ‘useCache’, it is important according to HERE when we use plugin version 2.1.0+. I use 2.5 for war-plugin

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-war-plugin</artifactId>
				<version>${war.plugin.version}</version>
            <executions>
               <!--add this phase so that we can replace the development js with production min.js. Without this, the replacer cannot find file to work on.-->
               <execution>
                  <id>prepare-war</id>
                  <phase>prepare-package</phase>
                  <goals>
                     <goal>exploded</goal>
                  </goals>
               </execution>
               <execution>
                  <id>default-war</id>
                  <phase>package</phase>
                  <goals>
                     <goal>war</goal>
                  </goals>
                  <configuration>
                     <warSourceDirectory>${project.build.directory}/${project.build.finalName}</warSourceDirectory>
                  </configuration>
               </execution>
            </executions>
				<configuration>
					<warName>snap</warName>
					<archive>
						<manifest>
							<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
						</manifest>
						<manifestEntries>
							<Implementation-Build>${maven.build.timestamp}</Implementation-Build>
						</manifestEntries>
					</archive>
					<webResources>
						<resource>
							<filtering>true</filtering>
							<directory>src/main/webapp</directory>
							<includes>
								<include>**/web.xml</include>
							</includes>
						</resource>
					</webResources>
					<!-- Don't exclude anything (i.e. include the simulation page) when 
						doing a local build. -->
					<warSourceExcludes>**/loginSimulation.jsp</warSourceExcludes>
               <!-- version 2.1.0+ copies resources twice which overrides replacements, we use this config to 'fix' this.-->
               <useCache>true</useCache>
            </configuration>
			</plugin>

When it comes to local profile:

               <!--in local build, we want to keep the develop js, so ignore the replace plugin. -->
               <plugin>
                  <groupId>com.google.code.maven-replacer-plugin</groupId>
                  <artifactId>replacer</artifactId>
                  <version>${maven.replacer.version}</version>
                  <executions>
                     <execution>
                        <phase>none</phase>
                        <goals>
                           <goal>replace</goal>
                        </goals>
                     </execution>
                  </executions>
               </plugin>

Hope this helps. Reference here.

setup java home java8 and mvn in yosemite

1. download and install java 8.

2. download and extract maven

3. create a .bash_profile under ~ (Users/yourname) if you do not have such a file (use ls -a to view all files including hidden ones). This file contains the options when you start your terminal.

4.  find your java_home first:

Java 7 and Java 8 are installed under /Library/Java/JavaVirtualMachines, and to get informations about the installed Java VM’s you can use the /usr/libexec/java_home command, which is a link/shortcut to /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home.

Some of the command to use are:

/usr/libexec/java_home -verbose will give you all installed versions with path.

/usr/libexec/java_home -v '1.6*' will give you JAVA_HOME of Java 6

/usr/libexec/java_home -v '1.7*' will give you JAVA_HOME of Java 7 (if installed)

/usr/libexec/java_home -v '1.8*' will give you JAVA_HOME of Java 8 (if installed)

And if /usr/libexec/java_home is called without any parameters you should get the current enabled (newest) Java version. If you don’t get the newest Java version when /usr/libexec/java_home is called, you can try a restart of your Mac, so that the system reload its current load of the /usr/bin/java link/shortcut to /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java

5. add following lines to the .bash_profile

export JAVA_HOME=your_java_home_path

export M2_HOME=your_maven_path

export PATH=$PATH:$M2_HOME/bin

6. now run java -version and mvn –version to check the installation.

maven override execution in profile

 

      <plugins>
         <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>gwt-maven-plugin</artifactId>
            <executions>
               <execution>
                  <id>pretty</id>
                  <goals>
                     <goal>compile</goal>
                  </goals>
                  <configuration>
                     <style>PRETTY</style>
                     <module>org.xxx.cdxxxip.xxx.ApplicationDebug</module>
                  </configuration>
               </execution>
               <!--default execution-->
               <execution>
                  <goals>
                     <goal>compile</goal>
                  </goals>
                  <configuration>
                     <style>${gwt.style}</style>
                     <extraJvmArgs>-Xms512m -Xmx1024m -Djava.awt.headless=true</extraJvmArgs>
                     <localWorkers>2</localWorkers>
                     <module>org.xxx.xxx.xxx.Application</module>
                     <!-- HTML Unit mode is needed for the GWT Test Cases so they don't run in manual (user input required) mode. -->
                     <!--<mode>htmlunit</mode>-->
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>

You can set the <phase> of the corresponding execution to something unknown, like none:

 <phase>none</phase> when overriding each execution. 

   <profiles>
      <profile>
         <id>local</id>
         <build>
            <plugins>
               <plugin>
                  <groupId>org.codehaus.mojo</groupId>
                  <artifactId>gwt-maven-plugin</artifactId>
                  <executions>
                     <execution>
                        <id>pretty</id>
                        <!--disable the this execution for local build-->
                        <phase>none</phase>
                     </execution>
                     <execution>
                        <goals>
                           <goal>compile</goal>
                        </goals>
                        <configuration>
                           <style>PRETTY</style>
                           <module>org.xxx.xxx.xxx.ApplicationLocal</module>
                        </configuration>
                     </execution>
                  </executions>
                  <configuration>
                     <extraJvmArgs>
                        <!--
                         Args must be on a single line, otherwise it puts carriage returns in the command.
                         This is being wrapped in CDATA to protect against IDE auto-formatting.
                        -->
                        <![CDATA[-Xms256m -Xmx512m -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:2956,suspend=n,server=y -Dfile.encoding=windows-1252]]>
                     </extraJvmArgs>

                     <!--add this to make local compilation faster but less-optimized, this sets the optimize level to 0. 9 is the max optimization value -->
                     <!--<draftCompile>true</draftCompile>-->
                  </configuration>
               </plugin>
            </plugins>
         </build>
      </profile>
   </profiles>