Teragrep DPL Command-Line Executor (pth_16)

Teragrep DPL Command-Line Executor is an utility to run DPL queries from command-line.

Features

• Allows running Teragrep queries from command-line.

Documentation

See the official documentation on docs.teragrep.com.

Limitations

Uses spark-submit.

Kafka saving requires spk_03 libraries to be present.

Requirements

Requires Spark installation to be available, including spark-submit binary.

Requires the following Hadoop configurations from a valid cluster to be available in HADOOP_CONF_DIR:

• core-site.xml

• hdfs-site.xml

• yarn-site.xml

Requires interpreter.json from zep_01 to be available.

Requires pth_10-shaded.jar to be available.

Uses spk_02 and hdp_03 libraries from the running cluster and they do not need to be locally available as long as cluster has them.

How to use

Usage

Connecting to Kerberos secured clusters requires you to set export KRB5CCNAME="/tmp/krb5cc_${UID}"; and then running kinit with valid credentials. You can use custom krb5.conf by setting export SPARK_SUBMIT_OPTS="-Djava.security.krb5.conf=custom-krb5.conf" before running.

Example without arguments

HADOOP_CONF_DIR=/opt/teragrep/hdp_03/etc/hadoop JAVA_HOME=/usr/lib/jvm/java-11 /opt/teragrep/spk_02/bin/spark-submit \
  --verbose \
  --deploy-mode client \
  --master yarn \
  --executor-memory 5G \
  --total-executor-cores 8 \
  --jars /opt/teragrep/pth_10/lib/pth_10-shaded.jar,/opt/teragrep/spk_03/lib/*.jar,local:/opt/teragrep/spk_02/jars/*,local:/opt/teragrep/hdp_03/share/hadoop/common/*,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar \
  --files interpreter.json \
  pth_16-jar-with-dependencies.jar

Results into

{"_time":"2026-01-27T13:37:00.000+02:00","_raw":"Welcome to Teragrep®"}

Custom query

HADOOP_CONF_DIR=/opt/teragrep/hdp_03/etc/hadoop JAVA_HOME=/usr/lib/jvm/java-11 /opt/teragrep/spk_02/bin/spark-submit \
  --verbose \
  --deploy-mode client \
  --master yarn \
  --executor-memory 5G \
  --total-executor-cores 8 \
  --jars /opt/teragrep/pth_10/lib/pth_10-shaded.jar,/opt/teragrep/spk_03/lib/*.jar,local:/opt/teragrep/spk_02/jars/*,local:/opt/teragrep/hdp_03/share/hadoop/common/*,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar \
  --files interpreter.json \
  pth_16-jar-with-dependencies.jar \
  '| makeresults count=1 | eval _raw="this is a custom query"'

Results into

{"_time":"2026-01-27T13:37:54.000+02:00","_raw":"this is a custom query"}

Results can be piped into a file

HADOOP_CONF_DIR=/opt/teragrep/hdp_03/etc/hadoop JAVA_HOME=/usr/lib/jvm/java-11 /opt/teragrep/spk_02/bin/spark-submit \
  --verbose \
  --deploy-mode client \
  --master yarn \
  --executor-memory 5G \
  --total-executor-cores 8 \
  --jars /opt/teragrep/pth_10/lib/pth_10-shaded.jar,/opt/teragrep/spk_03/lib/*.jar,local:/opt/teragrep/spk_02/jars/*,local:/opt/teragrep/hdp_03/share/hadoop/common/*,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar,local:/opt/teragrep/hdp_03/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.367.jar \
  --files interpreter.json \
  pth_16-jar-with-dependencies.jar \
  '| makeresults count=1 | eval _raw="this is a custom query"' | tee output.json

Results into

{
  "_time": "2026-01-27T13:54:24.000+02:00",
  "_raw": "this is a custom query"
}

Contributing

You can involve yourself with our project by opening an issue or submitting a pull request.

Contribution requirements:

1. All changes must be accompanied by a new or changed test. If you think testing is not required in your pull request, include a sufficient explanation as why you think so.

2. Security checks must pass

3. Pull requests must align with the principles and values of extreme programming.

4. Pull requests must follow the principles of Object Thinking and Elegant Objects (EO).

Read more in our Contributing Guideline.

Contributor License Agreement

Contributors must sign Teragrep Contributor License Agreement before a pull request is accepted to organization’s repositories.

You need to submit the CLA only once. After submitting the CLA you can contribute to all Teragrep’s repositories.