Skip to content

Environment settings

Prior to running any code related to desired actions (such as data extraction, data ingestion or thin API extraction) one must establish NXCALS environment by setting up certain properties as follows:

Action/Property service.url kafka.producer.bootstrap.servers spark.servers.url
Data extraction X
Data ingestion X X
Thin API extraction X

Examples of code snippets for setting these values can be found in our "Quick guide" for Java examples project under this link.

Depending on the environment PRO or TESTBED the following values must be used:

Property Value
service.url https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094
kafka.producer.bootstrap.servers cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092
spark.servers.url cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000
Property Value
service.url https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093
kafka.producer.bootstrap.servers cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092
spark.servers.url cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000

Please note that in case of Python (using pip install nxcals or Spark bundle) the service.url setting will come from the bundle itself (it is set in spark-defaults.conf). In order to connect to other environments (like TESTBED) you have to set it in the method spark_session_builder.get_or_create(service_url=...).

Required ports to open in firewall on the local machine

In certain cases NXCALS requires that firewall is open for some number of ports. That applies to running Spark in YARN client mode. By default NXCALS is using the following ports:

spark.driver.port                       5001
spark.blockManager.port                 5101
spark.ui.port                           5201

Important

In certain situations proposed values may conflict with ports which are arleady in use. In that case they can be changed to the new ones, selected from the following ranges:

5001-5100  ->  Spark driver
5101-5200  ->  Block manager
5201-5300  ->  Driver UI

Hint

Issuing the telnet command:

 telnet [domainname or ip] [port] 

allows to test connectivity to a remote host on the given port, for example:

telnet yourmachine.cern.ch 5001
A failed connection, accompanied by an error message, most likely indicates a firewall issue.