With my most populous post “Basic Checkpoint Gaia CLI Commands (Tips and Tricks)“, I would like to collect some more advanced troubleshooting commands used in my daily work into this post. Actually, some of commands are not only for Checkpoint Gaia, it will be for SPLAT or IPSO platform as well. This post will keep updating as soon as I have something new.
a. Use the Gaia portal. Network Management -> Arp -> Proxy ARP b. Use the command line (in Gaia): add arp proxy ipv4-address 172.16.0.8 interface eth0 real-ipv4-address 172.16.0.22 Actually the GAIA command above convert it automatically to a file called local.arp c. Use the command line (in expert mode): Then insert the information directly to /opt/CPsuite-R76/fw1/conf/local.arp echo “172.16.0.8 00:0c:29:f1:b7:74 172.16.0.22” >> $FWDIR/conf/local.arp
Verify the changes after a policy push with command “fw ctl arp”:
[[email protected]:0]# fw ctl arp (10.9.3.21) at 00-1c-7f-32-cc-15 (10.9.3.53) at 00-1c-7f-32-cc-15 (10.9.3.35) at 00-1c-7f-32-cc-15 (10.9.3.26) at 00-1c-7f-32-cc-15 (10.9.3.29) at 00-1c-7f-32-cc-15 (10.9.3.80) at 00-1c-7f-32-cc-15 (191.24.11.13) at 00-1c-7f-33-07-ae interface 191.24.11.116 (10.9.3.25) at 00-1c-7f-32-cc-15 (10.9.3.61) at 00-1c-7f-32-cc-15 (10.9.3.28) at 00-1c-7f-32-cc-15 (10.9.3.24) at 00-1c-7f-32-cc-15 (10.9.3.27) at 00-1c-7f-32-cc-15 FW-GAIA> show arp proxy all IP Address MAC Address / Interface Real IP Address
lists all dropped packets in real time gives an explanation why the packet is dropped
4. TCPDUMP
tcpdump port 257 , <– on the firewall, this will allow you to see if the logs are passing from the firewall to the manager, and what address they are heading to.
tcpdump -i WAN.15 <- to capture everything on this interface
tcpdump -i eth1.16 icmp <– to capture just PINGs on this interface
tcpdump -i Mgmt -vvv -s0 -w tcpdumpfile.log <– this captures the FULL packets to a file usefull for wireshark the -s0 stops the files being shortened
tcpdump -i INT port 67 <– view dhcp requests
tcpdump -eP -nni any host 10.9.4.30 <-disable both name and service port resolution while performing a capture, by using the -nn option; -e Print the link-level header on each dump line. This can be used, for example, to print MAC layer addresses for protocols such as Ethernet and IEEE 802.11. -p–no-promiscuous-mode.
tcpdump -i any <- any can be used to tell tcpdump to listen on all interfaces
tcpdump -n <- disable to lookup and translate hostnames and ports.
fw monitor -e ‘accept host(192.168.1.12);’ <– Show packets with IP 192.168.1.12 as SRC or DST
fw monitor -e ‘accept src=192.168.1.12 and dst=192.168.3.3;’ <–Show all packets from 192.168.1.12 to 192.168.3.3
fw monitor -pi ipopt_strip -e ‘accept udpport(53);’ <–Show UDP port 53 (DNS) packets, pre-in position is before ‘ippot_strip’
fw monitor -m O -e ‘accept udp and (sport>1023 or dport>1023);’ <– Show UPD traffic from or to unprivileged ports, only show post-out
fw monitor -e ‘accept net(192.168.1.0,24) and tracert;’ <–Show Windows traceroute (ICMP, TTL<30) from and to network 192.168.1.0/24
fw monitor -v 23 -e ‘accept tcpport(80);’ <–Show Capture web traffic for VSX virtual system ID 23
fw monitor -e ‘accept ip_p=50 and ifid=0;’ <–Show all ESP (IP protocol 50) packets on the interface with the ID 0. (List interfaces and corresponding IDs with fw ctl iflist)
srfw monitor -o output_file.cap <–Show traffic on a SecuRemote/SecureClient client into a file. srfw.exe is in $SRDIR/bin (C:Program FilesCheckPointSecuRemotebin)
6. VPN tu
vpn tu or vpn tunnelutil
********** Select Option **********
(1) List all IKE SAs (2) List all IPsec SAs (3) List all IKE SAs for a given peer (GW) or user (Client) (4) List all IPsec SAs for a given peer (GW) or user (Client) (5) Delete all IPsec SAs for a given peer (GW) (6) Delete all IPsec SAs for a given User (Client) (7) Delete all IPsec+IKE SAs for a given peer (GW) (8) Delete all IPsec+IKE SAs for a given User (Client) (9) Delete all IPsec SAs for ALL peers and users (0) Delete all IPsec+IKE SAs for ALL peers and users
Checkpoint SK60080 displays some solutions to resolve excessive disk consumption on SPLAT/Gaia/IPSO/Lunix OS system. Here are some helpful commands: a. df -h (view the partition table and its associated utilization) b. du -h –max-depth=1 /opt | sort -n -r (examine disk space utilization at directory-level) c. ls -1 $FWDIR/conf/db_versions/repository/ | wc -l (check the number of database revisions on a Security Management server) d. ls -l $RTDIR/distrib/* | wc -l (counts the number of records) e. evstop & evstart (Stop / start the Eventia / SmartEvent) f. rm -r $RTDIR/distrib/* (Purge this directory of stale records)
g. ls -lR /var/log/dump/usermode/ (Find and delete old core dump files)
h. ls -lR /var/crash/ (Find and delete old core dump files)
i. rm $FWDIR/log/2009*.log* (removes all old log files for year 2009)
8. Connections
CP-1>fw tab -t connections -s HOST NAME ID #VALS #PEAK #SLINKS localhost connections 8158 77 948 179
Note: The NAME Id is the actual table number. The VALS colum is the current number of connections that are in the connections table at the time the command was run. The PEAK number is the max number of connections that have been recorded since the last reboot. The SLINKS table is a table of symbolic link that point to the real connection entry. There are usually 4 symbolic links per connection. This way no matter which direction the packet comes, there will be an entry for it. There is more to it than that, but that is the general idea.
CP-1>fw ctl pstat
System Capacity Summary: Memory used: 8% (62 MB out of 696 MB) – below watermark Concurrent Connections: 0% (79 out of 24900) – below watermark Aggressive Aging is in detect mode Hash kernel memory (hmem) statistics: Total memory allocated: 71303168 bytes in 17408 (4096 bytes) blocks using 1 pool Total memory bytes used: 9703728 unused: 61599440 (86.39%) peak: 18891512 Total memory blocks used: 2665 unused: 14743 (84%) peak: 4705 Allocations: 198489371 alloc, 0 failed alloc, 198382561 free System kernel memory (smem) statistics: Total memory bytes used: 117769900 peak: 120093268 Total memory bytes wasted: 996590 Blocking memory bytes used: 2530356 peak: 2557584 Non-Blocking memory bytes used: 115239544 peak: 117535684 Allocations: 433810 alloc, 28 failed alloc, 432937 free, 0 failed free vmalloc bytes used: 114086588 expensive: no Kernel memory (kmem) statistics: Total memory bytes used: 56103032 peak: 66020104 Allocations: 198922588 alloc, 28 failed alloc 198815489 free, 0 failed free External Allocations: 0 for packets, 0 for SXL Cookies: 90753187 total, 0 alloc, 0 free, 7839 dup, 2107678 get, 160176 put, 91154457 len, 0 cached len, 0 chain alloc, 0 chain free Connections: 231169 total, 7807 TCP, 4665 UDP, 182351 ICMP, 36346 other, 0 anticipated, 3 recovered, 79 concurrent, 948 peak concurrent Fragments: 0 fragments, 0 packets, 0 expired, 0 short, 0 large, 0 duplicates, 0 failures NAT: 80509/0 forw, 5266/0 bckw, 85750 tcpudp, 16 icmp, 10440-949656 alloc Sync: Version: new Status: Able to Send/Receive sync packets Sync packets sent: total : 864451, retransmitted : 0, retrans reqs : 15, acks : 1826 Sync packets received: total : 3614413, were queued : 30, dropped by net : 15 retrans reqs : 0, received 11745 acks retrans reqs for illegal seq : 0 dropped updates as a result of sync overload: 0 Callback statistics: handled 11588 cb, average delay : 1, max delay : 5
9. Check Point SecureXL
To enable SecureXL, run the command: CP[admin]# fwaccel on
To disable SecureXL, run the command: CP[admin]# fwaccel off
Note: The fwaccel off command is not persistent and SecureXL will be enabled again after a reboot of the system. SecureXL can be permanently disabled through the CPconfig utility.
To check the number of accelerated connection and other SecureXL statistics: CP[admin]# netstat -f To check the number of accelerated SA (VPN traffic): CP[admin]# netstat -s To check overall SecureXL statistics: CP[admin]# fwaccel stat
10.View Checkpoint Log from CLI
expert mode
fw log -n | morefw log -n -f | https
or
normal mode without pipe
11. Revision Control Versions Location on Management Server
[[email protected]]# cd /opt/CPsuite-R75.20/fw1/conf/db_versions/repository/
Steps to Installation a Jumbo Hotfix for R77.20 on Cluster Environment: a. install a hotfix on standby cluster member (CP2) then reboot it b. failover from active cluster member (CP1) to standby cluster (CP2) after standby cluster finished rebooting c. install hotfix on CP1 and reboot it.
[[email protected]:0]# md5sum Check_Point_R77.20.linux.tgz d788583cf44389b83b0dd6990cb53f63 Check_Point_R77.20.linux.tgz [[email protected]:0]# tar -zxvf Check_Point_R77.20.linux.tgz Actions/ Actions/cpconfig Actions/CheckPackage Actions/CRSValidator Actions/GetPa …… [[email protected]:0]# ./UnixInstallScript *********************************************************** Welcome to Check Point R77_20_JUMBO_HF installation *********************************************************** Verifying installation environment for R77_20_JUMBO_HF…Done! The following components will be installed: * R77_20_JUMBO_HF Installation program is about to stop all Check Point Processes. Do you want to continue (y/n) ? y Stopping Check Point Processes…Done! Installing Security Gateway / Security Management R77_20_JUMBO_HF…Done! Installing GAIA R77_20_JUMBO_HF…Done! Installing Performance Pack R77_20_JUMBO_HF…Done! Installing Mobile Access R77_20_JUMBO_HF…Done! ************************************************************************ Package Name Status ———— —— Security Gateway / Security Management R77_20_JUMBO_HF Succeeded GAIA R77_20_JUMBO_HF Succeeded Performance Pack R77_20_JUMBO_HF Succeeded Mobile Access R77_20_JUMBO_HF Succeeded ************************************************************************ Installation program completed successfully. Do you wish to reboot your machine (y/n) ? y Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! Broadcast message from admin (pts/2) (Mon Oct 26 16:37:44 2015): The system is going down for reboot NOW! [[email protected]:0]#
15. SSH Timeout Solutions
a. Increasing the timeout set inactivity-timeout 720
Disown the process with this command, specifying the PID:
disown 27524
Closing, or having the SSH session end due to timeout will no longer send a hangup to this process, since it is no longer a child process of the SSH session.
A new SSH session or console session can be started later, and the process can be killed manually to stop it.
This example is creating cyclic debug files, 10 files, with a maximum of 50000Kb. Again, the PID is displayed, and the output of the command is piped to the text file ‘nohup.out’.
The SSH session can be ended with “exit” or timed out, and the hangup sent to this child process will be ignored, the debug will continue running until we log in again and manually kill the PID.
16. ClusterXL Troubleshooting
16.1 Force a failover
This creates a pnote (problem notification) that is in problem state at current cluster member and force a failover to another member:
cphaprob -d fail -s problem -t 0 register
Verify it’s in problem state with
cphaprob stat
and
cphaprob -i list
(you should see ‘fail’ in problem state)
Once you’ve finished your testing, run these two to reset it:
cphaprob -d fail -s ok reportcphaprob -d fail unregister
16.2 cphaprob commands and troubleshooting ClustXL Problem
FW-CP2 is fine. But FW-CP1 shows problem on the clustxl status.
[[email protected]:0]# cphaprob -a if Required interfaces: 5 Required secured interfaces: 1 eth1 UP non sync(non secured), multicast eth2 UP sync(secured), multicast Mgmt UP non sync(non secured), multicast eth3 UP non sync(non secured), multicast (eth3.106 ) eth3 UP non sync(non secured), multicast (eth3.102 ) Virtual cluster interfaces: 6 eth1 2.13.11.60 eth2 10.1.90.14 Mgmt 10.1.72.14 eth3.104 10.1.104.14 eth3.106 10.1.106.14 eth3.102 10.1.102.14
FW-CP1> cphaprob -i list Built-in Devices: Device Name: Interface Active Check Current state: problem Device Name: HA Initialization Current state: OK Device Name: Recovery Delay Current state: OK Registered Devices: Device Name: Synchronization Registration number: 0 Timeout: none Current state: OK Time since last report: 64196.3 sec Device Name: Filter Registration number: 1 Timeout: none Current state: OK Time since last report: 63492.1 sec Device Name: cphad Registration number: 2 Timeout: none Current state: OK Time since last report: 2.68138e+06 sec Device Name: fwd Registration number: 3 Timeout: none Current state: OK Time since last report: 2.68137e+06 sec Device Name: routed Registration number: 4 Timeout: none Current state: OK Time since last report: 62898.8 sec
Usually it was caused by the connection between firewall interface port and switch port. UDP port 8116 will help us to find out which one is not sending the keep-alive packets:
Cluster Control Protocol (CCP) runs on UDP port 8116, and allows cluster members to report their own states and learn about the states of other members, by sending keep-alive packets (applies only to ClusterXL clusters). Also CCP keeps cluster member sync state.
Following tcpdump shows cluster member 1 (00:00:00:00:fe:00) and cluster member 2 (00:00:00:00:fe:01) both are sending 8116 CCP packets. That is normal. If you only see one sending, you will have to check another one’s switch port vlan configuration. You may miss one vlan on switch trunk port, which has happened to me.
17. Permanent Change Global Kernel Parameters Value Global kernel parameters exist to control (customize) the behavior of Security Gateway (kernel parameters are located in $FWDIR/boot/modules/fw*mod* kernel modules).
This control (customization) can be done on-the-fly using the fw ctl set int command (change takes effect immediately). However, the value of the kernel parameter returns to its default value after a reboot. At times, it may be required to control (customize) the behavior of Security Gateway permanently. In addition, it is necessary for some kernel parameters to be changed upon boot. fwkern.conf file is the one which holds all those kernel parameters value. If it is not existing in your system, you will need to create it manually.
The Security Gateway must be rebooted after any change in the $FWDIR/boot/modules/fwkern.conf file.
