Environment settings
Prior to running any code related to desired actions (such as data extraction, data ingestion or thin API extraction) one must establish NXCALS environment by setting up certain properties as follows:
| Action/Property | service.url | kafka.producer.bootstrap.servers | spark.servers.url |
|---|---|---|---|
| Data extraction | X | ||
| Data ingestion | X | X | |
| Thin API extraction | X |
Examples of code snippets for setting these values can be found in our "Quick guide" for Java examples project under this link.
Depending on the environment PRO or TESTBED the following values must be used:
| Property | Value |
|---|---|
| service.url | https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094 |
| kafka.producer.bootstrap.servers | cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092 |
| spark.servers.url | nxcals-spark-thin-api-lb.cern.ch:14500,cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000,cs-513-nxcalsstr1.cern.ch:15000,cs-513-nxcalsstr2.cern.ch:15000,cs-513-nxcalsstr3.cern.ch:15000,cs-513-nxcalsstr4.cern.ch:15000 |
| Property | Value |
|---|---|
| service.url | https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093 |
| kafka.producer.bootstrap.servers | cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092 |
| spark.servers.url | nxcals-spark-thin-api-testbed-lb.cern.ch:14500,cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000 |
Please note that in case of Python (using pip install nxcals or Spark bundle) the service.url setting will come from the bundle itself (it is set in spark-defaults.conf). In order to connect to other environments (like TESTBED) you have to set it in the method spark_session_builder.get_or_create(service_url=...).
In order to simplify the configuration of services, we provide predefined values for environments such as PRO and TESTBED. These values are defined in both Python and Java files and can be easily reused in your project, helping avoid manual entry.
# Example of accessing predefined values in Python
from nxcals.api.custom.config import EnvironmentSettings
# Getting settings for PRO environment
pro_services = EnvironmentSettings.PRO.SERVICES
pro_kafka_servers = EnvironmentSettings.PRO.KAFKA_SERVERS
pro_spark_servers = EnvironmentSettings.PRO.SPARK_SERVERS
# Similarly, values for TESTBED
testbed_services = EnvironmentSettings.TESTBED.SERVICES
testbed_kafka_servers = EnvironmentSettings.TESTBED.KAFKA_SERVERS
testbed_spark_servers = EnvironmentSettings.TESTBED.SPARK_SERVERS
import cern.nxcals.api.custom.config.EnvironmentSettings;
public class EnvironmentSettingsSnippet {
public static void getEnvironmentSettings() {
// Getting settings for PRO environment
String proServices = EnvironmentSettings.PRO.SERVICES;
String proKafkaServers = EnvironmentSettings.PRO.KAFKA_SERVERS;
String proSparkServers = EnvironmentSettings.PRO.SPARK_SERVERS;
// Similarly, values for TESTBED environment
String testbedServices = EnvironmentSettings.TESTBED.SERVICES;
String testbedKafkaServers = EnvironmentSettings.TESTBED.KAFKA_SERVERS;
String testbedSparkServers = EnvironmentSettings.TESTBED.SPARK_SERVERS;
}
}
Required ports to open in firewall on the local machine
In certain cases NXCALS requires that firewall is open for some number of ports. That applies to running Spark in YARN client mode. By default NXCALS is using the following ports:
spark.driver.port 5001
spark.blockManager.port 5101
spark.ui.port 5201
Important
In certain situations proposed values may conflict with ports which are arleady in use. In that case they can be changed to the new ones, selected from the following ranges:
5001-5100 -> Spark driver
5101-5200 -> Block manager
5201-5300 -> Driver UI
Hint
Issuing the telnet command:
telnet [domainname or ip] [port]
allows to test connectivity to a remote host on the given port, for example:
telnet yourmachine.cern.ch 5001