Skip to content

Quick guide

This quick guide walks you through the process of creating applications that use various NXCALS APIs.

What you will build

You will build 4 applications that will make use of:

  • NXCALS Ingestion API to send messages with data to NXCALS Testbed environment in an asynchronous way.
  • NXCALS Extraction API to retrieve some sample data from NXCALS PRO using Spark and to fetch metadata related to variable hierarchies.
  • NXCALS Backport API to demonstrate retrieval of NXCALS PRO timeseries data and metadata using methods backward compatible with the old CALS API.
  • NXCALS Extraction Thin API to retrieve some data from NXCALS PRO using the thin Spark server API. It requires to have an RBAC token present, it does not require Kerberos token.

How to complete this guide

You can start from scratch and complete each step or you can bypass basic setup steps that are already familiar to you. To start from scratch, move on to one of the followings step-by-step guides:

To skip the basics and run the examples code immediately, do the following:

  1. Install sources on your machine using git clone: 
    git clone ssh://git@gitlab.cern.ch:7999/acc-logging-team/nxcals-examples.git
  2. cd into nxcals-examples and build all the modules: 
    ./gradlew clean build -Duser.name="your.username" -Duser.password="your.password"

    Please provide values for "your.username" and "your.password". These are required
by *extraction-api-thin-examples* module which requires RBAC login to work.
  3. Obtain Kerberos ticket and check its validity (Kerberos software must be installed): 
    kinit
klist
  4. Run examples from all the modules: 
    ./gradlew run -Duser.name="your.username" -Duser.password="your.password"
    or individually: 
    cd extraction-api-examples; ../gradlew run; cd ..

cd ingestion-api-examples; ../gradlew run; cd ..

cd backport-api-examples; ../gradlew runTimeseries; ../gradlew runMetadata; cd ..

cd extraction-api-thin-examples; ../gradlew run -Duser.name="your.username" -Duser.password="your.password"; cd ..

At this stage an output from the programs executions should be generated.

Changing username/environment

Need to run examples as another user or in another environment?

Setting environment (required)

For using of using NXCALS APIs, main classes of the example applications must contain a neccessary minimum bootstrap (enclosed in a static block):

static {
    // NXCALS Testbed
    System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");

    // Required for ingestion
    System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092");

    // Required for Thin API
    System.setProperty("spark.servers.url", "nxcals-spark-thin-api-testbed-lb:14500,cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000");
}

static {
    // NXCALS PRO
    System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");

    // Required for ingestion
    System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092");

    // Required for Thin API
    System.setProperty("spark.servers.url", "nxcals-spark-thin-api-lb:14500,cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000");
}

Setting a user (optional)

NXCALS uses Kerberos to authenticate it's users. If you require to impersonate another user/service account, some "adjustment" is required before running the examples.

  1. kerberos.keytab system property (and spark.kerberos.keytab for YARN) may need to be adjusted and point to the keytab file location
  2. kerberos.principal (and spark.kerberos.principal for YARN) must be equal to username of the user/service account that you need to impersonate

from the provided bootstrap template it is sufficient to uncomment the following lines and supply it with a appropriate user credential information 

    //System.setProperty("kerberos.principal", "nxcalsuser");
    //System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");
    //System.setProperty("spark.kerberos.principal", "nxcalsuser");
    //System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");

More info at: Kerberos keytab file generation

Another way of executing the application as another user is through the usage of RBAC authentication.

Below one can find a complete code snippet with required settings for user called "nxcalsuser" pointing to NXCALS PRO and Testbed environments:

    static {
        System.setProperty("logging.config", "classpath:log4j2.yml");

        // Uncomment in order to overwrite the default security settings. 
        // System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
        // System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");

        // Uncomment to provide principal and path to your keytab, by default the process acquires that info
        // from the locally cached kerberos ticket 
        System.setProperty("kerberos.principal", "nxcalsuser");
        System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // This is also needed in Spark on Yarn:
        System.setProperty("spark.kerberos.principal", "nxcalsuser");
        System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // NXCALS Testbed
        System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");
    }

    static {
        System.setProperty("logging.config", "classpath:log4j2.yml");

        // Uncomment in order to overwrite the default security settings.
        // System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
        // System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");

        // Uncomment to provide principal and path to your keytab, by default the process acquires that info
        // from the locally cached kerberos ticket
        System.setProperty("kerberos.principal", "nxcalsuser");
        System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // This is also needed in Spark on Yarn:
        System.setProperty("spark.kerberos.principal", "nxcalsuser");
        System.setProperty("spark.kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // NXCALS PRO
        System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");
    }

Configuration file

Building dependencies for Gradle or CBNG

Take note that for using JavaApi to access NXCALS data you should include our umbrella library nxcals-extraction-starter as dependency in your project. The jar is located in our artifactory repository http://artifactory.cern.ch/ds-release-local/cern/nxcals/nxcals-extraction-starter

Another required library is nxcals-hadoop-pro-config containing configutation for our production Hadoop installation (which is equally valid for NXCALS Testbed).

Examples of dependency configurations used for the project can be found below:

plugins
{
    id 'java-library'
}
apply plugin:'application'
mainClassName = "cern.myproject.DataAccessExample"

configurations.all {
    resolutionStrategy {
        eachDependency { DependencyResolveDetails details ->
            if (details.requested.group.startsWith('com.fasterxml.jackson')) {
                details.useVersion fasterxmlJacksonVersion
            }
        }
   }
// Depending on your gradle version you might have to do those exclusions (prior to gradle 5.x it did not honor the exclusions from POMs).
      exclude group: "org.springframework.boot", module: "spring-boot-starter-logging"
      exclude group: "javax-validation", module: "validation-api"
      exclude group: "org.slf4j", module: "slf4j-log4j12"
      exclude group: "log4j", module: "log4j"
      exclude group: "log4j", module: "apache-log4j-extras"
      exclude group: "ch.qos.logback", module: "logback-classic"
}

dependencies {
    api group: 'cern.nxcals', name: 'nxcals-extraction-starter', version: nxcalsVersion
    api group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion

    api group: 'org.springframework.boot', name: 'spring-boot-starter', version: springBootVersion
    api group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: springBootVersion
    api group: 'org.springframework.boot', name: 'spring-boot-starter-log4j2', version: springBootVersion

    api group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: log4jVersion
    api group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: log4jVersion

    //Required for Yaml in Log4j2
    api group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: fasterxmlJacksonVersion
    api group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: fasterxmlJacksonVersion

    //Required for CBNG as it does not support force statements from above that work in pure gradle
    api group: 'com.google.code.gson', name: 'gson', version: gsonVersion

    compileOnly group: 'org.projectlombok', name: 'lombok', version: lombokVersion
}

<?xml version="1.0" encoding="UTF-8"?>
<!-- !!! IMPORTANT: for CBNG configuration remove build.gradle files !!! -->
<products xmlns="http://cbng.cern.ch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://cbng.cern.ch http://bewww.cern.ch/ap/devops/cbng/cbng-library.xsd" >

<product name="java-data-access" version="0.1.0" groupId="cern.nxcals.nxcals-examples">
        <desc>Example code for interaction with the NXCALS service</desc>
        <href/>

        <dependencies>
            <dep product="nxcals-extraction-starter" />
            <dep product="nxcals-hadoop-pro-config" />

            <dep product="spring-expression" />

            <!--All the Spring Boot starters depend on spring-boot-starter-logging, which uses Logback by default.-->
            <!--For using Log4j2, we exclude spring-boot-starter-logging and add spring-boot-starter-log4j2 dependency.-->
            <dep product="spring-boot-starter" />
            <dep product="spring-boot-starter-validation" />
            <dep product="spring-boot-starter-log4j2" />

            <dep product="slf4j-api" />
            <dep product="log4j-1.2-api" />
            <dep product="log4j-slf4j-impl" />

            <dep product="jackson-dataformat-yaml" />
            <dep product="gson" />
        </dependencies>

        <exclusions>
            <dep groupId="org.springframework.boot" artifactId="spring-boot-starter-logging"/>
            <dep groupId="ch.qos.logback" artifactId="logback-classic"/>
            <dep groupId="org.slf4j" artifactId="slf4j-log4j12"/>
        </exclusions>
    </product>
</products>

Summary

Well done! At this point you should be able to build and run our 4 applications demonstrating usage of various NXCALS APIs.