Skip to content

Quick guide

This quick guide walks you through the process of creating applications that use various NXCALS APIs.

What you will build

You will build 4 applications that will make use of:

  • NXCALS Ingestion API to send messages with data to NXCALS Testbed environment in an asynchronous way.
  • NXCALS Extraction API to retrieve some sample data from NXCALS PRO using Spark and to fetch metadata related to variable hierarchies.
  • NXCALS Backport API to demonstrate retrieval of NXCALS PRO timeseries data and metadata using methods backward compatible with the old CALS API.
  • NXCALS Extraction Thin API to retrieve some data from NXCALS PRO using the thin Spark server API. It requires to have an RBAC token present, it does not require Kerberos token.

How to complete this guide

You can start from scratch and complete each step or you can bypass basic setup steps that are already familiar to you. To start from scratch, move on to one of the followings step-by-step guides:

To skip the basics and run the examples code immediately, do the following:

  1. Install sources on your machine using git clone:
    git clone ssh://git@gitlab.cern.ch:7999/acc-logging-team/nxcals-examples.git
    
  2. cd into nxcals-examples and build all the modules:
    ./gradlew clean build -Duser.name="your.username" -Duser.password="your.password"
    
    Please provide values for "your.username" and "your.password". These are required
    by *extraction-api-thin-examples* module which requires RBAC login to work.
    
  3. Obtain Kerberos ticket and check its validity (Kerberos software must be installed):
    kinit
    klist
    
  4. Run examples from all the modules:
    ./gradlew run -Duser.name="your.username" -Duser.password="your.password"
    
    or individually:
    cd extraction-api-examples; ../gradlew run; cd ..
    
    cd ingestion-api-examples; ../gradlew run; cd ..
    
    cd backport-api-examples; ../gradlew runTimeseries; ../gradlew runMetadata; cd ..
    
    cd extraction-api-thin-examples; ../gradlew run -Duser.name="your.username" -Duser.password="your.password"; cd ..
    

At this stage an output from the programs executions should be generated.

Changing username/environment

Need to run examples as another user or in another environment?

Setting environment (required)

For using of using NXCALS APIs, main classes of the example applications must contain a neccessary minimum bootstrap (enclosed in a static block):

static {
    // NXCALS Testbed
    System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");

    // Required for ingestion
    System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092");

    // Required for Thin API
    System.setProperty("spark.servers.url", "cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000");
}
static {
    // NXCALS PRO
    System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");

    // Required for ingestion
    System.setProperty("kafka.producer.bootstrap.servers", "cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092");

    // Required for Thin API
    System.setProperty("spark.servers.url", "cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000");
}

Setting a user (optional)

NXCALS uses Kerberos to authenticate it's users. If you require to impersonate another user/service account, some "adjustment" is required before running the examples.

  1. kerberos.keytab system property may need to be adjusted and point to the keytab file location
  2. kerberos.principal must be equal to username of the user/service account that you need to impersonate

from the provided bootsrap template it is sufficient to uncomment the following lines and supply it with a apprropriate informatin

    //System.setProperty("kerberos.principal", "nxcalsuser");
    //System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");

More info at: Kerberos keytab file generation

Another way of executing the application as another user is through the usage of RBAC authentication.

Below one can find a complete code snippet with required settings for user called "nxcalsuser" pointing to NXCALS PRO and Testbed environments:

    static {
        System.setProperty("logging.config", "classpath:log4j2.yml");

        // Uncomment in order to overwrite the default security settings. 
        //System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
        //System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");

        // Uncomment to provide principal and path to your keytab, by default the process acquires that info
        // from the locally cached kerberos ticket 
        System.setProperty("kerberos.principal", "nxcalsuser");
        System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // NXCALS Testbed
        System.setProperty("service.url", "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093");
    }
    static {
        System.setProperty("logging.config", "classpath:log4j2.yml");

        // Uncomment in order to overwrite the default security settings.
        //System.setProperty("javax.net.ssl.trustStore", "/opt/nxcalsuser/nxcals_cacerts");
        //System.setProperty("javax.net.ssl.trustStorePassword", "nxcals");

        // Uncomment to provide principal and path to your keytab, by default the process acquires that info
        // from the locally cached kerberos ticket
        System.setProperty("kerberos.principal", "nxcalsuser");
        System.setProperty("kerberos.keytab", "/opt/nxcalsuser/.keytab");

        // NXCALS PRO
        System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094");
    }

Configuration file

Building dependencies for Gradle or CBNG

Take note that for using JavaApi to access NXCALS data you should include our umbrella library nxcals-extraction-starter as dependency in your project. The jar is located in our artifactory repository http://artifactory.cern.ch/ds-release-local/cern/nxcals/nxcals-extraction-starter

Another required library is nxcals-hadoop-pro-config containing configutation for our production Hadoop installation (which is equally valid for NXCALS Testbed).

Examples of dependency configurations used for the project can be found below:

apply plugin:'application'
mainClassName = "cern.myproject.DataAccessExample"

configurations.all {
    resolutionStrategy {
        eachDependency { DependencyResolveDetails details ->
            if (details.requested.group.startsWith('com.fasterxml.jackson')) {
                details.useVersion fasterxmlJacksonVersion
            }
        }
   }
// Depending on your gradle version you might have to do those exclusions (prior to gradle 5.x it did not honor the exclusions from POMs).
      exclude group: "org.springframework.boot", module: "spring-boot-starter-logging"
      exclude group: "javax-validation", module: "validation-api"
      exclude group: "org.slf4j", module: "slf4j-log4j12"
      exclude group: "log4j", module: "log4j"
      exclude group: "log4j", module: "apache-log4j-extras"
      exclude group: "ch.qos.logback", module: "logback-classic"
}

dependencies {
    compile group: 'cern.nxcals', name: 'nxcals-extraction-starter', version: nxcalsVersion
    compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion


    compile group: 'org.springframework.boot', name: 'spring-boot-starter', version: springBootVersion
    compile group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: springBootVersion
    compile group: 'org.springframework.boot', name: 'spring-boot-starter-log4j2', version: springBootVersion

    compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: log4jVersion
    compile group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: log4jVersion

    //Required for Yaml in Log4j2
    compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: fasterxmlJacksonVersion
    compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: fasterxmlJacksonVersion

    compileOnly group: 'org.projectlombok', name: 'lombok', version: lombokVersion
}
<?xml version="1.0" encoding="UTF-8"?>

<!-- !!! IMPORTANT: for CBNG configuration remove build.gradle files !!! -->
<products>
    <product name="java-data-access" version="0.1.0" directory="accsoft/nxcals-examples/java-data-access">
        <desc>Example code for interaction with the NXCALS service</desc>
        <href/>

        <dependencies>
            <dep product="nxcals-extraction-starter" />
            <dep product="nxcals-hadoop-pro-config" />

            <dep product="spring-expression" />

            <!--All the Spring Boot starters depend on spring-boot-starter-logging, which uses Logback by default.-->
            <!--For using Log4j2, we exclude spring-boot-starter-logging and add spring-boot-starter-log4j2 dependency.-->
            <dep product="spring-boot-starter" />
            <dep product="spring-boot-starter-validation" />
            <dep product="spring-boot-starter-log4j2" />

            <dep product="slf4j-api" />
            <dep product="log4j-1.2-api" />
            <dep product="log4j-slf4j-impl" />

            <dep product="jackson-dataformat-yaml" />
            <dep product="gson" />
        </dependencies>

        <exclusions>
            <dep groupId="org.springframework.boot" artifactId="spring-boot-starter-logging"/>
            <dep groupId="ch.qos.logback" artifactId="logback-classic"/>
            <dep groupId="org.slf4j" artifactId="slf4j-log4j12"/>
        </exclusions>
    </product>
</products>

Summary

Well done! At this point you should be able to build and run our 4 applications demonstrating usage of various NXCALS APIs.