Layer4

Advanced tools: playing with Java Native Access

Posted by in Java

This post results from a recent deep diving in the source code of Elasticsearch, which uses JNA mainly for memory management when configuring the mlockall. I will present you how to use JNA in a very simple example: how to check the user who has launched the JVM. Note Remember that when you are thinking about solution using OS native calls, you must deal and depend with platform librairies. So use it carefully and only for specific needs. First of all, we need JNA: <dependency> <groupId>net.java.dev.jna</groupId> <artifactId>jna</artifactId> <optional>true</optional> <version>3.2.3</version> </dependency>…read more

OS monitoring with… Java

Posted by in DevOps

Sometimes it may be useful to get system information like the usage of a disk or the available network interfaces. For instance, Elasticsearch use this kind of tools in order to display at startup time some infos about open file descriptors or the size of the direct memory available for the JVM. The aim is not to replace a real system monitoring agent, but to guide the user to take advantage of the product by configuring it properly. Sigar is an open source project from Hyperic which provides a portable…read more

Find and kill slow running queries in MongoDB

Posted by in DevOps, NoSQL

In Mongo, or more generally in any data storage engine, queries or updates that take longer than expected to run can be caused by many reasons: – Slow network – Wrong schema design (we all have seen the famous all-in-one table…) – Wrong database design (“let’s store 100To of data in a standalone mongod!”) – Bad partitioning (Hbase table with 200 regions with 2MB of data) – Lack of useful indexes – No statistics – Incorrect hardware sizing (in 99.999%, the memory…) – Missing or wrong configuration (aka the default…read more

JSON pretty print with… Spring Boot

Posted by in Java

I wrote a post a few months ago about a pretty-print hack for JAX-RS (https://blog.layer4.fr/2015/02/json-pretty-print-with-jax-rsjackson/). Here is a version using Spring Boot. All the stuff is located in a @Configuration class extending WebMvcConfigurerAdapter: import java.io.IOException; import java.util.List; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Configuration; import org.springframework.http.converter.HttpMessageConverter; import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter; import org.springframework.web.context.request.RequestAttributes; import org.springframework.web.context.request.RequestContextHolder; import org.springframework.web.context.request.ServletRequestAttributes; import org.springframework.web.servlet.config.annotation.EnableWebMvc; import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter; import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jackson.core.util.DefaultPrettyPrinter; import com.fasterxml.jackson.databind.ObjectMapper; @Configuration @EnableWebMvc public class WebConfiguration extends WebMvcConfigurerAdapter { @Autowired private ObjectMapper mapper; @Override public void extendMessageConverters(List<HttpMessageConverter<?>> converters) { converters.replaceAll(c -> { if (c instanceof MappingJackson2HttpMessageConverter) { MappingJackson2HttpMessageConverter converter =…read more

0

JSON pretty print with JAX-RS/Jackson

Posted by in Java

If you have already worked with ElasticSearch, you may have used a very useful companion when dealing with your curl: the “?pretty” parameter which allows you to see the JSON response well-formated. If not, have a look here, you’ll thank me later: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/common-options.html It can be usefull too to have this kind of feature in your JAX-RS project. And this is “pretty” simple (…). All you have to do is create a class extending JacksonJsonProvider: import java.io.IOException; import java.io.OutputStream; import javax.ws.rs.core.Context; import org.apache.cxf.jaxrs.ext.MessageContext; import com.fasterxml.jackson.core.JsonEncoding; import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jackson.databind.ObjectMapper; import…read more

0

High performance RSS/Atom parsing

Posted by in Data, Java

Parsing RSS feeds is very easy in Java. Several libs exist to get the job done: feed4j, rssowl, Apache Abdera and many others. But the most commly used is ROME. ROME is a set of RSS and Atom Utilities for Java. It makes it easy to work in Java with most syndication formats: RSS 0.9x, 1.0, 2.0 and Atom 0.3, 1.0. Reading RSS from a source is dead-simple, you need these dependencies: <!– Rome Atom+RSS –> <dependency> <groupId>net.java.dev.rome</groupId> <artifactId>rome</artifactId> <version>1.0.0</version> </dependency> <dependency> <groupId>net.java.dev.rome</groupId> <artifactId>rome-fetcher</artifactId> <version>1.0.0</version> </dependency> and a piece of…read more

0

How to kill Hadoop jobs matching a pattern?

Posted by in DevOps

Today, I had to kill a list of jobs (45) running on my Hadoop cluster. Ok, let’s have a look to the docs http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#job But wait a minute… No, Hadoop knows the “kill” command, but not the “pkill”… One solution is: import java.io.IOException; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.CommandLineParser; import org.apache.commons.cli.HelpFormatter; import org.apache.commons.cli.Options; import org.apache.commons.cli.ParseException; import org.apache.commons.cli.PosixParser; import org.apache.commons.lang.ArrayUtils; import org.apache.commons.lang.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapred.JobClient; import org.apache.hadoop.mapred.JobStatus; import org.apache.hadoop.mapred.RunningJob; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class PKill { private final static Logger LOGGER = LoggerFactory.getLogger(PKill.class); private static void printUsage(Options options) { HelpFormatter usageFormatter…read more

0

Pooling a Thrift client

Posted by in Java

Thrift is an interface definition language that is used to define and create services for numerous languages. The Thrift stack relies on protocols (TBinaryProtocol, TCompactProtocol…) and transports (TSocket, TFileTransport…). But, since the transport layer is essentially a wrapper on a socket or a file, Thrift is NOT thread-safe. Like other resources not thread-safe, you have the choice: work with a costly locking algo, create each time a new connection to the resource or think about pool your connections. Locking is the simpliest way to share a not thread-safe resource, Java…read more

0

A best Spring AsyncRestTemplate!

Posted by in Java

I wrote an article on the famous Spring’s RestTemplate some months ago (https://blog.layer4.fr/2013/02/24/a-best-spring-resttemplate/). Since the last version of Spring 4, currently 4.0.0.M3, or the BUILD-SNAPSHOT for the most courageous, you can use a custom RestTemplate in an async way: the AsyncRestTemplate API. Why should I use that? The purpose of this API is to allow Java applications to easily execute HTTP requests and asynchronously process the HTTP responses. In non async mode, the code will block until the response is fully received. In async mode, the code will continue and…read more

4

Myrrix, the REST-ified Mahout for real-time recommandations

Posted by in Data

Myrrix is a complete, real-time, scalable clustering and recommender system, evolved from Apache Mahout. The full Myrrix system uses two components: a Computation Layer and one or many Serving Layers. While the Computation Layer computes the large machine learning models needed by the Serving Layer, the Serving Layer is a Java HTTP server application. This server serves user requests in real-time, making recommendations and receiving new input via a REST API. Many instances of the Serving Layer may be run for horizontal scalability. In stand-alone mode, one Serving Layer instance…read more

0