Quick guide
This quick guide walks you through the process of creating applications that use various NXCALS APIs.
What you will build
You will build 4 applications that will make use of:
- NXCALS Ingestion API to send messages with data to NXCALS Testbed environment in an asynchronous way.
- NXCALS Extraction API to retrieve some sample data from NXCALS PRO using Spark and to fetch metadata related to variable hierarchies.
- NXCALS Backport API to demonstrate retrieval of NXCALS PRO timeseries data and metadata using methods backward compatible with the old CALS API.
- NXCALS Extraction Thin API to retrieve some data from NXCALS PRO using the thin Spark server API. It requires to have an RBAC token present, it does not require Kerberos token.
How to complete this guide
You can start from scratch and complete each step or you can bypass basic setup steps that are already familiar to you. To start from scratch, move on to one of the followings step-by-step guides:
- NXCALS data ingestion
- NXCALS data extraction
- NXCALS backward compatible data retrieval
- NXCALS data extraction via thin Spark server
To skip the basics and run the examples code immediately, do the following:
- Install sources on your machine using git clone:
git clone ssh://git@gitlab.cern.ch:7999/acc-logging-team/nxcals-examples.git
- cd into
nxcals-examples
and build all the modules:./gradlew clean build -Duser.name="your.username" -Duser.password="your.password"
Please provide values for "your.username" and "your.password". These are required by *extraction-api-thin-examples* module which requires RBAC login to work.
- Obtain Kerberos ticket and check its validity (Kerberos software must be installed):
kinit klist
- Run examples from all the modules:
or individually:
./gradlew run -Duser.name="your.username" -Duser.password="your.password"
cd extraction-api-examples; ../gradlew run; cd .. cd ingestion-api-examples; ../gradlew run; cd .. cd backport-api-examples; ../gradlew runTimeseries; ../gradlew runMetadata; cd .. cd extraction-api-thin-examples; ../gradlew run -Duser.name="your.username" -Duser.password="your.password"; cd ..
At this stage an output from the programs executions should be generated.
Changing username/environment
Need to run examples as another user or in another environment?
Setting environment (required)
For using of using NXCALS APIs, main classes of the example applications must contain a neccessary minimum bootstrap (enclosed in a static block):
static {
// NXCALS Testbed
System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");
// Required for ingestion
System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092");
// Required for Thin API
System.setProperty("spark.servers.url", "nxcals-spark-thin-api-testbed-lb:14500,cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000");
}
static {
// NXCALS PRO
System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");
// Required for ingestion
System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092");
// Required for Thin API
System.setProperty("spark.servers.url", "nxcals-spark-thin-api-lb:14500,cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000");
}
Setting a user (optional)
NXCALS uses Kerberos to authenticate it's users. If you require to impersonate another user/service account, some "adjustment" is required before running the examples.
- kerberos.keytab system property (and spark.kerberos.keytab for YARN) may need to be adjusted and point to the keytab file location
- kerberos.principal (and spark.kerberos.principal for YARN) must be equal to username of the user/service account that you need to impersonate
from the provided bootstrap template it is sufficient to uncomment the following lines and supply it with a appropriate user credential information
//System.setProperty("kerberos.principal", "nxcalsuser");
//System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");
//System.setProperty("spark.kerberos.principal", "nxcalsuser");
//System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");
More info at: Kerberos keytab file generation
Another way of executing the application as another user is through the usage of RBAC authentication.
Below one can find a complete code snippet with required settings for user called "nxcalsuser" pointing to NXCALS PRO and Testbed environments:
static {
System.setProperty("logging.config", "classpath:log4j2.yml");
// Uncomment in order to overwrite the default security settings.
// System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
// System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");
// Uncomment to provide principal and path to your keytab, by default the process acquires that info
// from the locally cached kerberos ticket
System.setProperty("kerberos.principal", "nxcalsuser");
System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");
// This is also needed in Spark on Yarn:
System.setProperty("spark.kerberos.principal", "nxcalsuser");
System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");
// NXCALS Testbed
System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");
}
static {
System.setProperty("logging.config", "classpath:log4j2.yml");
// Uncomment in order to overwrite the default security settings.
// System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
// System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");
// Uncomment to provide principal and path to your keytab, by default the process acquires that info
// from the locally cached kerberos ticket
System.setProperty("kerberos.principal", "nxcalsuser");
System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");
// This is also needed in Spark on Yarn:
System.setProperty("spark.kerberos.principal", "nxcalsuser");
System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");
// NXCALS PRO
System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");
}
Configuration file
Building dependencies for Gradle or CBNG
Take note that for using JavaApi to access NXCALS data you should include our umbrella library nxcals-extraction-starter as dependency in your project. The jar is located in our artifactory repository http://artifactory.cern.ch/ds-release-local/cern/nxcals/nxcals-extraction-starter
Another required library is nxcals-hadoop-pro-config containing configutation for our production Hadoop installation (which is equally valid for NXCALS Testbed).
Examples of dependency configurations used for the project can be found below:
plugins
{
id 'java-library'
}
apply plugin:'application'
mainClassName = "cern.myproject.DataAccessExample"
configurations.all {
resolutionStrategy {
eachDependency { DependencyResolveDetails details ->
if (details.requested.group.startsWith('com.fasterxml.jackson')) {
details.useVersion fasterxmlJacksonVersion
}
}
}
// Depending on your gradle version you might have to do those exclusions (prior to gradle 5.x it did not honor the exclusions from POMs).
exclude group: "org.springframework.boot", module: "spring-boot-starter-logging"
exclude group: "javax-validation", module: "validation-api"
exclude group: "org.slf4j", module: "slf4j-log4j12"
exclude group: "log4j", module: "log4j"
exclude group: "log4j", module: "apache-log4j-extras"
exclude group: "ch.qos.logback", module: "logback-classic"
}
dependencies {
api group: 'cern.nxcals', name: 'nxcals-extraction-starter', version: nxcalsVersion
api group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion
api group: 'org.springframework.boot', name: 'spring-boot-starter', version: springBootVersion
api group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: springBootVersion
api group: 'org.springframework.boot', name: 'spring-boot-starter-log4j2', version: springBootVersion
api group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: log4jVersion
api group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: log4jVersion
//Required for Yaml in Log4j2
api group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: fasterxmlJacksonVersion
api group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: fasterxmlJacksonVersion
//Required for CBNG as it does not support force statements from above that work in pure gradle
api group: 'com.google.code.gson', name: 'gson', version: gsonVersion
compileOnly group: 'org.projectlombok', name: 'lombok', version: lombokVersion
}
<?xml version="1.0" encoding="UTF-8"?>
<!-- !!! IMPORTANT: for CBNG configuration remove build.gradle files !!! -->
<products xmlns="http://cbng.cern.ch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://cbng.cern.ch http://bewww.cern.ch/ap/devops/cbng/cbng-library.xsd" >
<product name="java-data-access" version="0.1.0" groupId="cern.nxcals.nxcals-examples">
<desc>Example code for interaction with the NXCALS service</desc>
<href/>
<dependencies>
<dep product="nxcals-extraction-starter" />
<dep product="nxcals-hadoop-pro-config" />
<dep product="spring-expression" />
<!--All the Spring Boot starters depend on spring-boot-starter-logging, which uses Logback by default.-->
<!--For using Log4j2, we exclude spring-boot-starter-logging and add spring-boot-starter-log4j2 dependency.-->
<dep product="spring-boot-starter" />
<dep product="spring-boot-starter-validation" />
<dep product="spring-boot-starter-log4j2" />
<dep product="slf4j-api" />
<dep product="log4j-1.2-api" />
<dep product="log4j-slf4j-impl" />
<dep product="jackson-dataformat-yaml" />
<dep product="gson" />
</dependencies>
<exclusions>
<dep groupId="org.springframework.boot" artifactId="spring-boot-starter-logging"/>
<dep groupId="ch.qos.logback" artifactId="logback-classic"/>
<dep groupId="org.slf4j" artifactId="slf4j-log4j12"/>
</exclusions>
</product>
</products>
Summary
Well done! At this point you should be able to build and run our 4 applications demonstrating usage of various NXCALS APIs.