java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected

While writing the unit java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected test cases for my
application I was getting the below mentioned exception during the execution of these test cases. Below is the  complete exception trace :

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
	at org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper.createCommon(AbstractMockContextWrapper.java:59)
	at org.apache.hadoop.mrunit.internal.mapreduce.MockMapContextWrapper.create(MockMapContextWrapper.java:77)
	at org.apache.hadoop.mrunit.internal.mapreduce.MockMapContextWrapper.(MockMapContextWrapper.java:68)
	at org.apache.hadoop.mrunit.mapreduce.MapDriver.getContextWrapper(MapDriver.java:167)
	at org.apache.hadoop.mrunit.mapreduce.MapDriver.getContext(MapDriver.java:198)
	at com.techidiocy.integratekeys.mapreduce.test.TestIntegrationKeysMapper.init(TestIntegrationKeysMapper.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO “TABLE_PARAMS” – Hive with Kite Morphlines

rdbms.exceptions.MappedDatastoreException: INSERT INTO “TABLE_PARAMS”

If you had read my previous post (Anatomy of a configuration file) where i hive_logo
described the flow of the application
on which I was working . In the last
command of my morphline I was
creating the hive table using the avro
schema which was created in the previous command.
As per the design of my application I
decided that i will store my avro schema
file in the local file system instead of saving it on HDFS. Everything was working fine as expected , but things started breaking when my avro schema file contains more than 4000 characters. Below is the complete exception which I got while Hive table creation :

org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO “TABLE_PARAMS” 
Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

java.io.IOException: can not read class parquet.format.PageHeader: null – Hive

While evaluating the Cloudera Kite Morphlines , java.io.IOException: can not read class parquet.format.PageHeader:
I came across this exception while reading the
data from the table.

java.io.IOException: can not read class
parquet.format.PageHeader: null
.

Before going ahead let me give you the background what I am trying to do here.
I am building an application where external client will upload input XML files and there corresponding XSDs ,once these files are uploaded a job will run that will unmarshall these XML files into Java objects , later on these these java objects will be passed to Drools Framework where validation and minor transformations will be performed on this data. During this Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Anatomy of a Command Builder with Example – Cloudera Kite Morphlines

Cloudera Kite Morphlines

In the last post we have seen the internals Cloudera Kite Morphlines
of a configuration file also known as
morphline. In this post we are going to
explore the actual code that does all the
job in the background. It doesn’t make a
difference whether you are using the in
built command (bundled with Cloudera Kite Morphlines SDK) or writing your own custom command , basic structure and semantics of all the commands are same.

All the commands in the Cloudera Kite Morphlines implements

org.kitesdk.morphline.api.CommandBuilder

interface. This interface contains 2 methods for which you have to provide the implementation in your CommandBuilder implementation. Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Anatomy of a Configuration File with Example- Cloudera Kite Morphlines

At the heart of the Cloudera Kite Morphlines Cloudera Kite Morphlines
is the configuration file that contains all of
your commands that you want to execute
as a part of your ETL process. In the last post we have seen the structure of a configuration file and how the commands are specified in the configuration file.

In Cloudera Kite Morphlines every configuration file ends with an extension of .conf , it is a little bit new and more specific to Morphlines. In this post we are going to dissect the configuration file that we had seen in the last post, we will see the flow of execution Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Cloudera Kite Morphlines Getting Started Example

Kite Morphlines development was initiated  Cloudera Kite Morphlines
as a part of Cloudera Search project and
later it was moved to Kite SDK to make it
more available to a wide range of users
and to invite contributions from the CDK
active community.Idea behind the Kite
Morphlines development is to streamline
the ETL processing , so that the time and
effort involved in Extraction , Transformation and Load of the huge data into Apache Solr, HBase, HDFS, Enterprise Data Warehouses can be reduced. Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Replace unicode characters from a Java String

It is a very common scenario that when your application depends on some external sources for input files and then it might be possible that these files may contain some special characters or might be different encoding format is used when this file was written in comparison to what is used for reading. Now for further processing it is important that these special characters should be removed. In this post we will see how to replace unicode characters from a Java String with their corresponding ascii values.

Before looking into the actual java code for replacing unicode characters , lets see what actually Unicode means.
As per the unicode.org definition. Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

java.lang.IllegalArgumentException: Wrong FS: expected: file:/// – Hadoop

Today while writing a java program to interact with the Hadoop File System(HDFS) I was getting the below mentioned exception (Wrong FS: expected: file:///).

Note : My hadoop cluster is running on Cloudera CDH 5.

i.e – Exception in thread “main” java.lang.IllegalArgumentException: Wrong FS: hdfs://ibc.techidiocy.com:8020/user/saurav/input.txt, expected: file:///

And below is the command that I was trying to execute in my local system to read a file from HDFS.

java -jar HDFSInteraction.jar hdfs://ibc.techidiocy.com:8020/user/saurav/input.txt

Here is the complete stack trace. Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Compound Indexes MongoDB Java Example

Compound Indexes MongoDB Java Example

Having a requirement on searchingIndexing in MongoDB multiple keys in a query or need the result in the sorted order on the basis of multiple keys this is where Compound Indexes come into the picture. If we look at our previous post example – Simple Index MongoDB Java Example,where we have created index on key username such that whenever we performs a query on userName key it always gives us the record in the optimal time and in sorted order of username. Now if I want my result to be in ascending order of age and descending order of username then in that case it won’t be optimized as results are not stored in the ascending order of age because we don’t have any index on age key. Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Single Field Indexes MongoDB Java Example

Indexing MongoDB Java Example

Indexing in MongoDB works in theIndexing in MongoDB same way as it is used to work in relational databases.It helps in the fast retrieval of documents available in MongDB . Having an index over a collection ensures that least numbers of documents are scanned to find the documents matching search criteria. Hence having an understanding of indexes is very important for the efficient performance of your application. In this post we will cover: Continue reading

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS