ETL

The OrientDB-ETL module is an amazing tool to move data from and to OrientDB by executing an ETL process. It's super easy to use. OrientDB ETL is based on the following principles:

• one configuration file in JSON format
• one Extractor is allowed to extract data from a source
• one Loader is allowed to load data to a destination
• multiple Transformers that transform data in pipeline. They receive something in input, do something, return something as output that will be processed as input by the next component

How ETL works

EXTRACTOR => TRANSFORMERS[] => LOADER

Example of a process that extract from a CSV file, apply some change, lookup if the record has already been created and then store the record as document against OrientDB database:

+-----------+-----------------------+-----------+
|           |              PIPELINE             |
+ EXTRACTOR +-----------------------+-----------+
|           |     TRANSFORMERS      |  LOADER   |
+-----------+-----------------------+-----------+
|   FILE   ==>  CSV->FIELD->MERGE  ==> OrientDB |
+-----------+-----------------------+-----------+

The pipeline, made of transformation and loading phases, can run in parallel by setting the configuration {"parallel":true}.

Installation

Starting from OrientDB v2.0 the ETL module will be distributed in bundle with the official release. If you want to use it, then follow these steps:

• Clone the repository on your computer, by executing:
  • git clone https://github.com/orientechnologies/orientdb-etl.git
• Compile the module, by executing:
  • mvn clean install
• Copy script/oetl.sh (or .bat under Windows) to $ORIENTDB_HOME/bin
• Copy target/orientdb-etl-2.0-SNAPSHOT.jar to $ORIENTDB_HOME/lib

Usage

$ cd $ORIENTDB_HOME/bin
$ ./oetl.sh config-dbpedia.json

Available Components

• Blocks
• Sources
• Extractors
• Transformers
• Loaders

Examples:

• Import from CSV to a Graph
• Import from JSON
• Import DBPedia
• Import from a DBMS
• Import from Parse (Facebook)