Skip to content

Backport API (CALS API via NXCALS)

We have implemented an indirection layer that enables using NXCALS via the old CALS API. We refer to the project as backport-api.

Deprecated API

Please note that some methods, public APIs or classes do not conform easily to the new system. We have marked a number of methods with @Deprecated to signal this situation.

While me might not remove these methods, you use them at your own risk. You might run into a series of issues, be it very low performance, unpredictable results or even inability to run at all.

Please avoid deprecated methods if possible and treat very seriously every warning about using a deprecated method in your project. Notify us if you consider a deprecated method to be absolutely essential to your workflow.

Dependencies

In order to start working with the backport-api first you have to import the required jars:

dependencies {
    (...)   
    compile group: 'cern.nxcals', name: 'nxcals-backport-api', version: nxcalsVersion

    compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion
    (...)
}

<?xml version="1.0" encoding="UTF-8"?>

<!-- !!! IMPORTANT: for CBNG configuration remove build.gradle files !!! -->
<products>
        (...)
        <dependencies>
            (...)
            <dep product="nxcals-backport-api" />

            <dep product="nxcals-hadoop-pro-config" />
            (...)
        </dependencies>
    </product>
</products>

Packages

We have renamed the java packages for all the public CALS classes:

  • cern.accsoft.cals.extr.client.service. -> cern.nxcals.api.backport.client.service.
  • cern.accsoft.cals.extr.domain. -> cern.nxcals.api.backport.client.service.

This allows you to effortlessly use both, the old api and the new api in a single project.

Authentication

Prerequisites:

  • The backport-api uses Kerberos for authentication. If you are unfamiliar with kerberos please consult getting started with kerberos.
  • You must request access to NXCALS as described here

Once you have your Kerberos and your account setup, you can continue.

There are two main ways of authenticating into NXCALS:

Authenticating with Kerberos shared state

NXCALS provides seamless Kerberos integration. In essence, all you need to do prior to running your code is to create your Kerberos token:

kinit

We will pick up the Kerberos token and authenticate you automatically. For as long as your token is valid, you can freely run backport-api methods. The advantage of this method is the simplicity. In addition, you can setup a cron job to renew the token indefinitely.

Authenticating with .keytab file

If you would like to authenticate using your .keytab file, you have to include the following statement in your code:

import org.apache.hadoop.security.UserGroupInformation;

(...)

String principle = YOUR PRINCIPLE;
String keytab = YOUR KEYTAB PATH;

UserGroupInformation.loginUserFromKeytab(principle, keytab);

Once this is done, you will have programmatically obtained a Kerberos token. The advantage of this solution is that you rely on your code to obtain the token. It is safer and more reliable.

Token expiry

Please note that kerberos token expires after a time. Regardless of your mode of authentication you must take care of renewing your token.

Configuration

From this point on all you need to configure is the service endpoint for NXCALS. This is done via a system property.

static {
    System.setProperty("service.url",
            "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19094");
}

static {
    System.setProperty("service.url",
            "https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093");
}

Temporary

We are working on simplifying this step. Soon you will not have to declare the service endpoints.

API differences

We have aimed at maintaining maximum API compatibility with old CALS API. As the systems differ deeply, there is a number of small and unavoidable changes in the Backport project, both in the syntax and logic. However, for the sake of this document the main concern is that we have replaced the old ServiceBuilder method:

public static ServiceBuilder newInstance(String app, String client, DataLocationPreferences prefs)

with: 

public static ServiceBuilder getInstance()

The old method lost its meaning, as there is no DataLocationPreference anymore, and the authentication is not performed via app and client name. The new method assumes reasonable defaults with respect to the Spark connection and abstracts the users completely from the spark layer. It is sufficient for most simple use-cases.

Other methods of Spark session creation are presented on dedicated page.

Putting it all together

At this point your project should look something like this:

package cern.nxcals.docs;
// NOTICE THE NEW PACKAGES ON IMPORTS

import cern.cmw.datax.ImmutableData;
import cern.cmw.datax.converters.JapcToDataxConverter;
import cern.japc.core.AcquiredParameterValue;
import cern.nxcals.api.backport.client.service.AcquiredParameterValuesService;
import cern.nxcals.api.backport.client.service.ServiceBuilder;
import cern.nxcals.api.backport.client.service.TimeseriesDataService;
import cern.nxcals.api.backport.domain.core.metadata.JapcParameterDefinition;
import cern.nxcals.api.backport.domain.core.metadata.Variable;
import cern.nxcals.api.backport.domain.core.timeseriesdata.TimeseriesDataSet;
import cern.nxcals.api.backport.domain.util.TimestampFactory;
import com.google.common.collect.ImmutableList;
import org.apache.hadoop.security.UserGroupInformation;

import java.io.IOException;
import java.sql.Timestamp;
import java.util.List;

class Main {
    static {
        // CONFIGURATION
        System.setProperty("service.url",
                "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19094");
    }

    public static void main(String[] args) throws IOException {
        // AUTHENTICATION

        String principle = "YOUR PRINCIPLE";
        String keytab = "YOUR KEYTAB PATH";

        UserGroupInformation.loginUserFromKeytab(principle, keytab);

        // below BACKPORT API using a default Spark configuration

        ServiceBuilder serviceBuilder = ServiceBuilder.getInstance();

        countValuesForVariable(serviceBuilder,
                "LTB.BCT60:INTENSITY",
                TimestampFactory.parseUTCTimestamp("2018-04-29 00:00:00.000"),
                TimestampFactory.parseUTCTimestamp("2018-04-30 00:00:00.000"));
        getValuesForParameter(serviceBuilder,
                "RADMON.PS-10",
                "ExpertMonitoringAcquisition",
                TimestampFactory.parseUTCTimestamp("2017-08-29 00:00:06.000"),
                TimestampFactory.parseUTCTimestamp("2017-08-29 00:00:07.000"));
    }

    private static void countValuesForVariable(ServiceBuilder serviceBuilder, String variableName,
                                               Timestamp startTime, Timestamp endTime) {

        TimeseriesDataService timeseriesDataService = serviceBuilder.createTimeseriesService();
        Variable variable = serviceBuilder.createMetaService()
                .getVariablesWithNameInListofStrings(ImmutableList.of(variableName)).iterator().next();

        TimeseriesDataSet data = timeseriesDataService.getDataInTimeWindow(variable, startTime, endTime);

        System.out.println("Number of values for variable: " + variable.getVariableName() + " size: " + data.size());
    }

    private static void getValuesForParameter(ServiceBuilder serviceBuilder, String deviceName, String propertyName,
                                              Timestamp startTime, Timestamp endTime) {
        JapcParameterDefinition japcParameterDefinition = JapcParameterDefinition.builder()
                .deviceName(deviceName).propertyName(propertyName).build();

        AcquiredParameterValuesService acqParamValService = serviceBuilder.createAcquiredParameterService();

        List<AcquiredParameterValue> acqParamValList = acqParamValService
                .getParameterValuesInTimeWindow(japcParameterDefinition, startTime, endTime);

        System.out.println("Values for parameter:");

        acqParamValList.forEach(p -> {
            ImmutableData immutableData = JapcToDataxConverter.toImmutableData(p);
            System.out.println(immutableData.getEntry("japc-parameter-value"));
        });
    }
}
Click to see expected application output... 
Number of values for variable: LTB.BCT60:INTENSITY size: 72000
Values for parameter: 
Name: japc-parameter-value
Type: ImmutableData
Value: 
[Name: __record_timestamp__
Type: Long
Value: 1503964806659000000

Name: __record_version__
Type: Long
Value: 0

Name: acqStamp
Type: Long
Value: 1503964806659000000

Name: class
Type: String
Value: "RadmonV6"

Name: cyclestamp
Type: Long
Value: 0

Name: device
Type: String
Value: "RADMON.PS-10"

Name: nxcals_entity_id
Type: Long
Value: 55461

Name: property
Type: String
Value: "ExpertMonitoringAcquisition"

Name: pt100Value
Type: Double
Value: 106.1]

For many more examples please consult the examples project, where you will find a fully configured and runnable classes with all the aspects of the backport api. All you need to do is to checkout the project and navigate to the module backport-api-examples and explore.