Environment settings
Prior to running any code related to desired actions (such as data extraction, data ingestion or thin API extraction) one must establish NXCALS environment by setting up certain properties as follows:
| Action/Property | service.url | kafka.producer.bootstrap.servers | spark.servers.url |
|---|---|---|---|
| Data extraction | X | ||
| Data ingestion | X | X | |
| Thin API extraction | X |
Examples of code snippets for setting these values can be found in our "Quick guide" for Java examples project under this link.
Depending on the environment PRO or TESTBED the following values must be used:
| Property | Value |
|---|---|
| service.url | https://cs-ccr-nxcals5.cern.ch:19093,https://cs-ccr-nxcals5.cern.ch:19094,https://cs-ccr-nxcals6.cern.ch:19093,https://cs-ccr-nxcals6.cern.ch:19094,https://cs-ccr-nxcals7.cern.ch:19093,https://cs-ccr-nxcals7.cern.ch:19094,https://cs-ccr-nxcals8.cern.ch:19093,https://cs-ccr-nxcals8.cern.ch:19094 |
| kafka.producer.bootstrap.servers | cs-ccr-nxcalsstr4.cern.ch:9092,cs-ccr-nxcalsstr5.cern.ch:9092,cs-ccr-nxcalsstr6.cern.ch:9092,cs-ccr-nxcalsstr7.cern.ch:9092 |
| spark.servers.url | cs-ccr-nxcals5.cern.ch:15000,cs-ccr-nxcals6.cern.ch:15000,cs-ccr-nxcals7.cern.ch:15000,cs-ccr-nxcals8.cern.ch:15000 |
| Property | Value |
|---|---|
| service.url | https://cs-ccr-testbed2.cern.ch:19093,https://cs-ccr-testbed2.cern.ch:19094,https://cs-ccr-testbed3.cern.ch:19093,https://cs-ccr-nxcalstbs1.cern.ch:19093,https://cs-ccr-nxcalstbs2.cern.ch:19093 |
| kafka.producer.bootstrap.servers | cs-ccr-nxcalstbs1.cern.ch:9092,cs-ccr-nxcalstbs2.cern.ch:9092,cs-ccr-nxcalstbs3.cern.ch:9092,cs-ccr-nxcalstbs4.cern.ch:9092 |
| spark.servers.url | cs-ccr-nxcalstbs2.cern.ch:15000,cs-ccr-nxcalstbs3.cern.ch:15000,cs-ccr-nxcalstbs4.cern.ch:15000 |
Please note that in case of Python (using pip install nxcals or Spark bundle) the service.url setting will come from the bundle itself (it is set in spark-defaults.conf). In order to connect to other environments (like TESTBED) you have to set it in the method spark_session_builder.get_or_create(service_url=...).
Required ports to open in firewall on the local machine
In certain cases NXCALS requires that firewall is open for some number of ports. That applies to running Spark in YARN client mode. By default NXCALS is using the following ports:
spark.driver.port 5001
spark.blockManager.port 5101
spark.ui.port 5201
Important
In certain situations proposed values may conflict with ports which are arleady in use. In that case they can be changed to the new ones, selected from the following ranges:
5001-5100 -> Spark driver
5101-5200 -> Block manager
5201-5300 -> Driver UI
Hint
Issuing the telnet command:
telnet [domainname or ip] [port]
allows to test connectivity to a remote host on the given port, for example:
telnet yourmachine.cern.ch 5001