This step-by-step-guide aims to walk you through the process of creating application demonstrating usage of NXCALS backport API.
What you will build
An application which retirves timeseries data from NXCALS using CALS backport API (backward compatible).
Create application
Create main class of the application:
public class MetadataExamples {
static {
// NXCALS PRO
System.setProperty("service.url", "https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19094");
}
public static void main(String[] args) {
}
}
Create some simple "helper" methods for retrieving Variable and printing datapoints:
private static Variable getVariable(String variableName) {
return ServiceBuilder.getInstance().createMetaService()
.getVariablesWithNameInListofStrings(ImmutableList.of(variableName)).iterator().next();
}
private static void printData(TimeseriesDataSet data) {
data.forEach(MetadataExamples::printDatapoint);
}
private static void printDatapoint(TimeseriesData dataPoint) {
try {
if (dataPoint == null) {
System.out.println("Absent data");
} else {
System.out.println(dataPoint.getStamp() + ": " + dataPoint.getDoubleValue());
}
} catch (NoSuchMethodException e) {
e.printStackTrace();
}
}
private static void printStatistics(VariableStatisticsSet statisticsSet) {
Iterator<VariableStatistics> iterator = statisticsSet.iterator();
while (iterator.hasNext()) {
final VariableStatistics next = iterator.next();
System.out.println("Values for variable : " + next.getVariableName() + " max: " + next.getMaxValue() + " min: " + next.getMinValue());
}
}
Create different methods for extracting data via CALS backport API:
public static void getDataAlignedToTimestamps() {
TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");
TimeseriesDataSet drivingSet = SparkTimeseriesDataSet.of(variable,
ImmutableList.of(
SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"), null),
SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:01:00.000"), null),
SparkTimeseriesData.of(TimestampFactory.parseUTCTimestamp("2018-06-19 00:02:00.000"), null))
);
TimeseriesDataSet data = timeseriesDataService.getDataAlignedToTimestamps(variable, drivingSet);
System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
printData(data);
}
public static void getDataInTimeWindow() {
TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");
TimeseriesDataSet data = timeseriesDataService.getDataInTimeWindow(variable,
TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000"));
System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
printData(data);
}
public static void getDataInTimeWindowAndLastDataPriorToTimeWindowWithinDefaultInterval() {
TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");
TimeseriesDataSet data = timeseriesDataService.getDataInTimeWindowOrLastDataPriorToWindowWithinDefaultInterval(variable,
TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000")
);
System.out.println("Values for variable : " + variable.getVariableName() + " size: " + data.size());
printData(data);
}
public static void getVariableStatisticsOverMultipleVariablesInTimeWindow() {
TimeseriesDataService timeseriesDataService = ServiceBuilder.getInstance().createTimeseriesService();
Variable variable = getVariable("PR.DCAFTINJ_1:INTENSITY");
VariableSet variableSet = new VariableSet(variable);
VariableStatisticsSet data = timeseriesDataService.getVariableStatisticsOverMultipleVariablesInTimeWindow(variableSet,
TimestampFactory.parseUTCTimestamp("2018-06-19 00:00:00.000"),
TimestampFactory.parseUTCTimestamp("2018-06-19 00:05:00.000")
);
printStatistics(data);
}
}
and add them to the main method:
public static void main(String[] args) {
getDataAlignedToTimestamps();
getDataInTimeWindow();
getDataInTimeWindowAndLastDataPriorToTimeWindowWithinDefaultInterval();
getVariableStatisticsOverMultipleVariablesInTimeWindow();
}
Build application
Prepare gradle.properties
file with the following content:
nxcalsVersion=1.3.26
springBootVersion=2.7.3
log4jVersion=2.17.2
fasterxmlJacksonVersion=2.13.4
Important
Make sure that you have the latest production version of NXCALS!
Prepare a "minimal" version of build.gradle
file and place it in the main directory of your application.
apply plugin: 'java'
task(runTimeseries, dependsOn: 'classes', type: JavaExec) {
main = 'cern.myproject.TimeseriesExamples'
classpath = sourceSets.main.runtimeClasspath
}
task(runMetadata, dependsOn: 'classes', type: JavaExec) {
main = 'cern.myproject.MetadataExamples'
classpath = sourceSets.main.runtimeClasspath
}
task run {
description 'Run different Backport API examples'
dependsOn 'runTimeseries'
dependsOn 'runMetadata'
tasks.findByName('runMetadata').mustRunAfter 'runTimeseries'
}
dependencies {
compile group: 'cern.nxcals', name: 'nxcals-backport-api', version: nxcalsVersion
compile group: 'cern.nxcals', name: 'nxcals-extraction-starter', version: nxcalsVersion
compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config', version: nxcalsVersion
compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: log4jVersion
compile group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: log4jVersion
//needed for log4j2 yaml file
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: fasterxmlJacksonVersion
compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: fasterxmlJacksonVersion
}
Important
Please note required dependencies for data extraction:
compile group: 'cern.nxcals', name: 'nxcals-backport-api'
compile group: 'cern.nxcals', name: 'nxcals-extraction-starter'
compile group: 'cern.nxcals', name: 'nxcals-hadoop-pro-config'
Build and run the application:
../gradlew build run
Click to see expected application output...
> Task :backport-api-examples:runTimeseries
Using shared state token to login
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Values for variable : PR.DCAFTINJ_1:INTENSITY size: 3
2018-06-19 02:00:00.0: 0.0
2018-06-19 02:01:00.0: 0.0
2018-06-19 02:02:00.0: 0.0
Values for variable : PR.DCAFTINJ_1:INTENSITY size: 238
2018-06-19 02:00:00.7: 8.809006690979004
2018-06-19 02:00:03.1: 0.0
2018-06-19 02:00:04.3: 0.0
2018-06-19 02:00:05.5: 0.0
2018-06-19 02:00:06.7: 0.0
2018-06-19 02:00:07.9: 0.0
2018-06-19 02:00:09.1: 0.0
2018-06-19 02:00:10.3: 0.0
2018-06-19 02:00:11.5: 0.0
2018-06-19 02:00:12.7: 0.0
2018-06-19 02:00:13.9: 0.0
2018-06-19 02:00:15.1: 0.0
2018-06-19 02:00:16.3: -0.07446788996458054
2018-06-19 02:00:17.5: 0.0
2018-06-19 02:00:18.7: 0.0
2018-06-19 02:00:19.9: 0.0
2018-06-19 02:00:21.1: 0.0
2018-06-19 02:00:22.3: 0.0
2018-06-19 02:00:23.5: 0.0
2018-06-19 02:00:24.7: 0.0
2018-06-19 02:00:25.9: 0.0
2018-06-19 02:00:27.1: 9.570235252380371
2018-06-19 02:00:29.5: 0.0
2018-06-19 02:00:30.7: 0.0
2018-06-19 02:00:31.9: 0.0
2018-06-19 02:00:33.1: 0.0
2018-06-19 02:00:34.3: 0.0
2018-06-19 02:00:35.5: 0.0
2018-06-19 02:00:36.7: 0.0
2018-06-19 02:00:37.9: 0.0
2018-06-19 02:00:39.1: 0.0
2018-06-19 02:00:40.3: 0.0
2018-06-19 02:00:41.5: 0.0
2018-06-19 02:00:42.7: -0.06538643687963486
2018-06-19 02:00:43.9: 0.0
2018-06-19 02:00:45.1: 0.0
2018-06-19 02:00:46.3: 0.0
2018-06-19 02:00:47.5: 0.0
2018-06-19 02:00:48.7: 0.0
2018-06-19 02:00:49.9: 0.0
2018-06-19 02:00:51.1: 0.0
2018-06-19 02:00:52.3: 0.0
2018-06-19 02:00:53.5: 9.14421558380127
2018-06-19 02:00:55.9: 0.0
2018-06-19 02:00:57.1: 0.0
2018-06-19 02:00:58.3: 0.0
2018-06-19 02:00:59.5: 0.0
2018-06-19 02:01:00.7: 0.0
2018-06-19 02:01:01.9: 0.0
2018-06-19 02:01:03.1: 0.0
2018-06-19 02:01:04.3: 0.0
2018-06-19 02:01:05.5: 0.0
...
2018-06-19 02:04:49.9: 0.0
2018-06-19 02:04:51.1: 9.281149864196777
2018-06-19 02:04:53.5: 0.0
2018-06-19 02:04:54.7: 0.0
2018-06-19 02:04:55.9: 0.0
2018-06-19 02:04:57.1: 0.0
2018-06-19 02:04:58.3: 0.0
2018-06-19 02:04:59.5: 0.0
Values for variable : PR.DCAFTINJ_1:INTENSITY max: 0 min: 0
Last value before 2018-06-19 00:05:00.000
2018-06-19 02:04:59.5: 0.0
Last value at most one day before 2019-04-09 00:00:00.000
2018-06-19 02:04:59.5: 0.0
Last value at most one day before now
Absent data
No data at most one day before now
Next value after 2018-06-19 00:05:00.000
2018-06-19 02:05:00.7: 0.0
Last value at most one day before 2018-06-19 00:05:00.000
2018-06-19 02:05:00.7: 0.0
> Task :backport-api-examples:runMetadata
Using shared state token to login
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Found Variable:PR.DCAFTINJ_1:INTENSITY of type UNDETERMINED and system CMW
Found Variable:DQQDS.B15R3.RQF.A34:ST_PWR_PERM of type UNDETERMINED and system WINCCOA
Found Variable:DQQDS.B15R3.RQF.A34:U_REF_N1 of type UNDETERMINED and system WINCCOA
Create Spark on YARN session (optional)
Depending on the resource requirements for the application, instead of default (local) spark session:
ServiceBuilder defaultServiceBuilder = ServiceBuilder.getInstance();
return defaultServiceBuilder;
}
SparkSessionFlavor initialProperties = SparkSessionFlavor.MEDIUM;
// Please replace "pro" with "testbed" when accessing NXCALS TESTBED
ServiceBuilder yarnServiceBuilderFromPredefinedProperties
= ServiceBuilder.getInstance(SparkProperties.remoteSessionProperties("MY_APP", initialProperties, "pro"));
More information about different ways of creating Spark session in Backport API can be found at this location.
Summary
You have successfully retrieved timeseries data from NXCALS using different CALS backport API methods.
See also
Other pages related to the presented example: