This step-by-step-guide aims to walk you through the process of creating application demonstrating usage of NXCALS backport API.

What you will build

An application which retirves timeseries data from NXCALS using CALS backport API (backward compatible).

Create application

Create main class of the application:

public class MetadataExamples {
        static {
            // NXCALS PRO
            System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19094");
        }

        public static void main(String[] args) {
        }
}

Create some simple "helper" methods for retrieving Variable and printing datapoints:

private static Variable getVariable(String variableName) {
        return ServiceBuilder.getInstance().createMetaService()
                .getVariablesWithNameInListofStrings(ImmutableList.of(variableName)).iterator().next();
    }

    private static void printData(TimeseriesDataSet data) {
        data.forEach(MetadataExamples::printDatapoint);
    }

    private static void printDatapoint(TimeseriesData dataPoint) {
        try {
            if (dataPoint == null) {
                System.out.println("Absent data");
            } else {
                System.out.println(dataPoint.getStamp() + ": " + dataPoint.getDoubleValue());
            }
        } catch (NoSuchMethodException e) {
            e.printStackTrace();
        }
    }

    private static void printStatistics(VariableStatisticsSet statisticsSet) {
        Iterator<VariableStatistics> iterator = statisticsSet.iterator();
        while (iterator.hasNext()) {
            final VariableStatistics next = iterator.next();
            System.out.println("Values for variable : " + next.getVariableName() + " max: " + next.getMaxValue() + " min: " + next.getMinValue());
        }
    }

Create different methods for extracting data via CALS backport API:

public static void getDataAlignedToTimestamps() {
        TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
        Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");

        TimeseriesDataSet drivingSet = SparkTimeseriesDataSet.of(variable,
                ImmutableList.of(
                        SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"), null),
                        SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:01:00.000"), null),
                        SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:02:00.000"), null))
        );

        TimeseriesDataSet data = timeseriesDataService.getDataAlignedToTimestamps(variable, drivingSet);

        System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
        printData(data);
    }

    public static void getDataInTimeWindow() {
        TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
        Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");

        TimeseriesDataSet data = timeseriesDataService.getDataInTimeWindow(variable,
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000"));

        System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
        printData(data);
    }

    public static void getDataInTimeWindowAndLastDataPriorToTimeWindowWithinDefaultInterval() {
        TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
        Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");

        TimeseriesDataSet data = timeseriesDataService.getDataInTimeWindowOrLastDataPriorToWindowWithinDefaultInterval(variable,
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000")
        );

        System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
        printData(data);
    }

    public static void getVariableStatisticsOverMultipleVariablesInTimeWindow() {
        TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
        Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");
        VariableSet variableSet = new VariableSet(variable);


        VariableStatisticsSet data = timeseriesDataService.getVariableStatisticsOverMultipleVariablesInTimeWindow(variableSet,
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
                TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000")
        );

        printStatistics(data);
    }
    }

and add them to the main method:

public static void main(String[] args) {

        getDataAlignedToTimestamps();
        getDataInTimeWindow();
        getDataInTimeWindowAndLastDataPriorToTimeWindowWithinDefaultInterval();
        getVariableStatisticsOverMultipleVariablesInTimeWindow();
    }

Build application

Prepare gradle.properties file with the following content:

nxcalsVersion=1.5.11

springBootVersion=2.7.16
log4jVersion=2.21.0
fasterxmlJacksonVersion=2.13.5

Important

Make sure that you have the latest production version of NXCALS!

Prepare a "minimal" version of build.gradle file and place it in the main directory of your application.

apply plugin: 'java'

task(runTimeseries, dependsOn: 'classes', type: JavaExec) {
    main = 'cern.myproject.TimeseriesExamples'
    classpath = sourceSets.main.runtimeClasspath
}

task(runMetadata, dependsOn: 'classes', type: JavaExec) {
    main = 'cern.myproject.MetadataExamples'
    classpath = sourceSets.main.runtimeClasspath
}

task run {
    description 'Run different Backport API examples'
    dependsOn 'runTimeseries'
    dependsOn 'runMetadata'
    tasks.findByName('runMetadata').mustRunAfter 'runTimeseries'
}

dependencies {
    compile group: 'cern.nxcals', name: 'nxcals-backport-api', version: nxcalsVersion

    compile group: 'cern.nxcals', name: 'nxcals-extraction-starter', version: nxcalsVersion
    compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion

    compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: log4jVersion
    compile group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: log4jVersion

    //needed for log4j2 yaml file
    compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: fasterxmlJacksonVersion
    compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: fasterxmlJacksonVersion
}

Important

Please note required dependencies for data extraction:

compile group: 'cern.nxcals', name: 'nxcals-backport-api'
compile group: 'cern.nxcals', name: 'nxcals-extraction-starter'
compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config'

Build and run the application:

../gradlew build run

Click to see expected application output...

> Task :backport-api-examples:runTimeseries
Using shared state token to login
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Values for variable : PR.DCAFTINJ_1:INTENSITY size: 3
2018-06-19 02:00:00.0: 0.0
2018-06-19 02:01:00.0: 0.0
2018-06-19 02:02:00.0: 0.0
Values for variable : PR.DCAFTINJ_1:INTENSITY size: 238
2018-06-19 02:00:00.7: 8.809006690979004
2018-06-19 02:00:03.1: 0.0
2018-06-19 02:00:04.3: 0.0
2018-06-19 02:00:05.5: 0.0
2018-06-19 02:00:06.7: 0.0
2018-06-19 02:00:07.9: 0.0
2018-06-19 02:00:09.1: 0.0
2018-06-19 02:00:10.3: 0.0
2018-06-19 02:00:11.5: 0.0
2018-06-19 02:00:12.7: 0.0
2018-06-19 02:00:13.9: 0.0
2018-06-19 02:00:15.1: 0.0
2018-06-19 02:00:16.3: -0.07446788996458054
2018-06-19 02:00:17.5: 0.0
2018-06-19 02:00:18.7: 0.0
2018-06-19 02:00:19.9: 0.0
2018-06-19 02:00:21.1: 0.0
2018-06-19 02:00:22.3: 0.0
2018-06-19 02:00:23.5: 0.0
2018-06-19 02:00:24.7: 0.0
2018-06-19 02:00:25.9: 0.0
2018-06-19 02:00:27.1: 9.570235252380371
2018-06-19 02:00:29.5: 0.0
2018-06-19 02:00:30.7: 0.0
2018-06-19 02:00:31.9: 0.0
2018-06-19 02:00:33.1: 0.0
2018-06-19 02:00:34.3: 0.0
2018-06-19 02:00:35.5: 0.0
2018-06-19 02:00:36.7: 0.0
2018-06-19 02:00:37.9: 0.0
2018-06-19 02:00:39.1: 0.0
2018-06-19 02:00:40.3: 0.0
2018-06-19 02:00:41.5: 0.0
2018-06-19 02:00:42.7: -0.06538643687963486
2018-06-19 02:00:43.9: 0.0
2018-06-19 02:00:45.1: 0.0
2018-06-19 02:00:46.3: 0.0
2018-06-19 02:00:47.5: 0.0
2018-06-19 02:00:48.7: 0.0
2018-06-19 02:00:49.9: 0.0
2018-06-19 02:00:51.1: 0.0
2018-06-19 02:00:52.3: 0.0
2018-06-19 02:00:53.5: 9.14421558380127
2018-06-19 02:00:55.9: 0.0
2018-06-19 02:00:57.1: 0.0
2018-06-19 02:00:58.3: 0.0
2018-06-19 02:00:59.5: 0.0
2018-06-19 02:01:00.7: 0.0
2018-06-19 02:01:01.9: 0.0
2018-06-19 02:01:03.1: 0.0
2018-06-19 02:01:04.3: 0.0
2018-06-19 02:01:05.5: 0.0

...


2018-06-19 02:04:49.9: 0.0
2018-06-19 02:04:51.1: 9.281149864196777
2018-06-19 02:04:53.5: 0.0
2018-06-19 02:04:54.7: 0.0
2018-06-19 02:04:55.9: 0.0
2018-06-19 02:04:57.1: 0.0
2018-06-19 02:04:58.3: 0.0
2018-06-19 02:04:59.5: 0.0
Values for variable : PR.DCAFTINJ_1:INTENSITY max: 0 min: 0
Last value before 2018-06-19 00:05:00.000
2018-06-19 02:04:59.5: 0.0
Last value at most one day before 2019-04-09 00:00:00.000
2018-06-19 02:04:59.5: 0.0
Last value at most one day before now
Absent data
No data at most one day before now
Next value after 2018-06-19 00:05:00.000
2018-06-19 02:05:00.7: 0.0
Last value at most one day before 2018-06-19 00:05:00.000
2018-06-19 02:05:00.7: 0.0

> Task :backport-api-examples:runMetadata
Using shared state token to login
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Found Variable:PR.DCAFTINJ_1:INTENSITY of type UNDETERMINED and system CMW
Found Variable:DQQDS.B15R3.RQF.A34:ST_PWR_PERM of type UNDETERMINED and system WINCCOA
Found Variable:DQQDS.B15R3.RQF.A34:U_REF_N1 of type UNDETERMINED and system WINCCOA

Create Spark on YARN session (optional)

Depending on the resource requirements for the application, instead of default (local) spark session:

ServiceBuilder defaultServiceBuilder = ServiceBuilder.getInstance();
    return defaultServiceBuilder;
    }

one can use one of the 3 predefined Spark properties for YARN:

SparkSessionFlavor initialProperties = SparkSessionFlavor.MEDIUM;

    // Please replace "pro" with "testbed" when accessing NXCALS TESTBED
    ServiceBuilder yarnServiceBuilderFromPredefinedProperties
            = ServiceBuilder.getInstance(SparkProperties.remoteSessionProperties("MY_APP", initialProperties, "pro"));

More information about different ways of creating Spark session in Backport API can be found at this location.

Summary

You have successfully retrieved timeseries data from NXCALS using different CALS backport API methods.