JunOS has strong flexibility on many features. One of them is logging. It support flexible logging options. This post summarizes some concepts I learned from my work and studying. 

1.Understand Juniper SRX logging Type:

1.1 System Logging

Junos OS supports configuring and monitoring of system log messages (also called syslog messages). You can configure files to log system messages and also assign attributes, such as severity levels, to messages. Reboot requests are recorded to the system log files, which you can view with the show log command. SRX Series devices can send system log messages from the control plane (Routing Engine) to one or more destinations. Destinations can include local files on the SRX Series device (because the SRX Series device is a syslog server), remote syslog servers, user terminals, and the system console.

admin@fw-1> show configuration system syslog
archive size 750k files 2;
user * {
    any emergency;
}
host 10.9.0.33 {
    any any;
    change-log none;
    interactive-commands none;
    explicit-priority;
}
host 10.9.8.52 {
    any any;
    source-address 10.9.8.20;
}
file messages {
    any critical;
    authorization info;
    explicit-priority;
}
file interactive-commands {
    interactive-commands error;
}
}


1.2 Traffic Logging (Event Mode)

You can use traffic logs to track usage patterns or troubleshoot issues for a specific policy. You can configure a policy so that traffic information is logged when a session begins (session-init) and/or closes (session-close). To generate traffic logs for multiple policies, you must configure each policy to log traffic information. You also must configure syslog messages with a severity level of info or any. In the default configuration, these messages and all other logging messages are sent to a local log file named messages.


admin@fw-1> show configuration system syslog
archive size 750k files 2;
user * {
    any emergency;
}
host 10.9.0.33 {
    any any;
    change-log none;
    interactive-commands none;
    explicit-priority;
}
host 10.9.8.52 {
    any any;
    source-address 10.9.8.20;
}
file messages {
    any critical;
    authorization info;
    explicit-priority;
}
file interactive-commands {
    interactive-commands error;
}
file traffic-create {
    any any;
    match RT_FLOW_SESSION_CREATE;
    structured-data;
}
file traffic-deny {
    any any;
    match RT_FLOW_SESSION_DENY;
}
file traffic-flow {
    user info;
    match RT_FLOW;
    archive size 1000k files 5 world-readable;
    structured-data;
}


admin@fw-1> show log ?
Possible completions:
  <[Enter]>            Execute this command
  <filename>           Name of log file
  IKELOG               Size: 270913, Last changed: Feb 15 2015
  PKITRACE             Size: 138153, Last changed: Oct 02 02:22:41
  PKITRACE.0.gz        Size: 98723, Last changed: Sep 27 05:12:14
  __jsrpd_commit_check__  Size: 6456, Last changed: Dec 21 2014
  appidd               Size: 0, Last changed: May 13 2014
  authd_libstats       Size: 0, Last changed: May 13 2014
  authd_profilelib     Size: 0, Last changed: May 13 2014
  authd_sdb.log        Size: 0, Last changed: May 13 2014
  authlib_jdhcpd_trace.log  Size: 0, Last changed: Jan 18 2015
  bin_messages         Size: 7, Last changed: May 13 2014
  chassisd             Size: 1173869, Last changed: Oct 01 22:54:45
  cosd                 Size: 98079, Last changed: Sep 20 11:36:47
  dcd                  Size: 251523, Last changed: Sep 21 14:25:57
  default-log-messages  Size: 612840, Last changed: Oct 02 02:19:39
  default-log-messages.0.gz  Size: 1027366, Last changed: Sep 20 18:45:01
  default-log-messages.1.gz  Size: 1323072, Last changed: Sep 20 18:30:00
  dfwc                 Size: 0, Last changed: May 13 2014
  e2e_events           Size: 239, Last changed: Sep 20 11:45:31
  eccd                 Size: 0, Last changed: May 13 2014
  ext/                 Last changed: May 13 2014
  flowc/               Last changed: May 13 2014
  fwauthd_chk_only     Size: 298, Last changed: Dec 21 2014
  ggsn/                Last changed: May 13 2014
  gprsd_chk_only       Size: 1335, Last changed: Dec 21 2014
  gres-tp              Size: 23569, Last changed: Sep 20 11:36:47
  group_db.log         Size: 0, Last changed: May 13 2014
  helplog              Size: 64, Last changed: Nov 17 2014
  hostname-cached      Size: 408, Last changed: Dec 21 2014
  httpd.log            Size: 1533, Last changed: Sep 20 18:36:17
  idpd                 Size: 0, Last changed: May 13 2014
  idpd.addver          Size: 185, Last changed: Sep 20 19:15:01
  idpd_err             Size: 208962, Last changed: Sep 20 19:15:11
  idpd_err.1           Size: 1048851, Last changed: Sep 20 18:55:14
  ifstraced            Size: 120, Last changed: Dec 21 2014
  indb                 Size: 967833, Last changed: Dec 21 2014
  install              Size: 3927, Last changed: Dec 21 2014
  interactive-commands  Size: 82, Last changed: Sep 21 14:25:52
  inventory            Size: 17170, Last changed: Sep 20 11:45:34
  ipfd                 Size: 97046, Last changed: Sep 28 10:01:08
  ipfd_chk_only        Size: 32, Last changed: Dec 21 2014
  jdhcpd_era_discover.log  Size: 8892, Last changed: Oct 01 20:07:42
  jdhcpd_era_discover.log.0  Size: 43387, Last changed: Aug 13 23:11:34
  jdhcpd_era_discover.log.1  Size: 25529, Last changed: Jun 19 14:49:47
  jdhcpd_era_discover.log.2  Size: 422808, Last changed: Apr 17 16:00:00
  jdhcpd_era_discover.log.3  Size: 0, Last changed: Jan 18 2015
  jdhcpd_era_solicit.log  Size: 595, Last changed: Sep 20 11:36:47
  jdhcpd_era_solicit.log.0  Size: 595, Last changed: Jul 19 13:01:48
  jdhcpd_era_solicit.log.1  Size: 595, Last changed: May 17 12:26:15
  jdhcpd_era_solicit.log.2  Size: 595, Last changed: Jan 18 2015
  jdhcpd_era_solicit.log.3  Size: 0, Last changed: Jan 18 2015
  jdhcpd_sdb.log       Size: 0, Last changed: Jan 18 2015
  jsrpd                Size: 841811, Last changed: Sep 28 10:01:17
  kmd                  Size: 369441, Last changed: Sep 20 18:36:26
  license              Size: 0, Last changed: May 13 2014
  license_subs_trace.log  Size: 16976, Last changed: Sep 20 11:36:47
  lsys-cpu-utilization-log  Size: 0, Last changed: May 13 2014
  mastership           Size: 13036, Last changed: Sep 20 11:36:47
  messages             Size: 687915, Last changed: Oct 02 02:26:11
  messages.0.gz        Size: 38283, Last changed: Sep 26 05:45:00
  messages.1.gz        Size: 38105, Last changed: Sep 24 22:15:00
  nsd_chk_only         Size: 1021282, Last changed: Sep 29 18:26:35
  nstraced             Size: 58027, Last changed: Sep 20 11:43:30
  nstraced_chk_only    Size: 370, Last changed: Mar 18 2015
  pcre_db.log          Size: 0, Last changed: May 13 2014
  pf                   Size: 1152, Last changed: Dec 21 2014
  pfed                 Size: 0, Last changed: May 13 2014
  pfed_jdhcpd_trace.log  Size: 0, Last changed: Jan 18 2015
  pgmd                 Size: 385, Last changed: Dec 21 2014
  pkid                 Size: 828994, Last changed: Dec 21 2014
  rexp_db.log          Size: 0, Last changed: May 13 2014
  rsi.1400.0118        Size: 4620227, Last changed: Jan 18 2015
  rsi_2015_02_04       Size: 4762354, Last changed: Feb 04 2015
  rtlogd               Size: 3952, Last changed: Sep 29 18:26:56
  smartd.trace         Size: 133439, Last changed: Oct 01 23:51:05
  traffic-create       Size: 9307887, Last changed: Oct 02 02:26:11
  traffic-create.0.gz  Size: 593738, Last changed: Oct 02 02:15:00
  traffic-create.1.gz  Size: 679624, Last changed: Oct 02 02:00:00
  traffic-deny         Size: 733722, Last changed: Oct 02 02:26:11
  traffic-deny.0.gz    Size: 30893, Last changed: Oct 02 02:00:00
  traffic-deny.1.gz    Size: 29997, Last changed: Oct 02 01:30:00
  traffic-flow         Size: 14043535, Last changed: Oct 02 02:26:11
  traffic-flow.0.gz    Size: 1110300, Last changed: Oct 02 02:15:00
  traffic-flow.1.gz    Size: 1194867, Last changed: Oct 02 02:00:00
  traffic-flow.2.gz    Size: 1223703, Last changed: Oct 02 01:45:00
  traffic-flow.3.gz    Size: 1205868, Last changed: Oct 02 01:30:00
  traffic-flow.4.gz    Size: 1196097, Last changed: Oct 02 01:15:01
  user                 Show recent user logins
  utmd-av              Size: 960, Last changed: Sep 20 11:36:47
  utmp                 Size: 0, Last changed: May 13 2014
  |                    Pipe through a command

1.3 Notes from Juniper KB:

System Logging Traffic Logging
SRX Branch Devices
SRX100
SRX110
SRX210
SRX220
SRX240
SRX550
SRX650
 KB16502  KB16509
SRX High-End Devices
SRX1400
SRX3400
SRX3600
SRX5600
SRX5800
 KB16502  KB16506

2. Understand Juniper SRX Logging Methods:

Control Plane and Data Plane

2.1 Control Plane Logging

The control plane logs have to do with events triggered by daemons on the control plane. This includes messages about the underlying hardware (chassisd), general-purpose messages (messages), and various protocol daemons like IDPD, appidd, and so on. Control plane logging is on by default to log locally, but you can override this with your own logfiles, syslog hosts, and criteria for different log messages. All logs are stored in the /var/log directory on the control plane. The configuration has been described at section 1.1

Services on the control plane:

  • Management Daemon (MGD):  Provides the interface between the UI components and the backend configuration and is responsible for acting on the Junos configuration to the system itself.
  • Routing Protocol Daemon (RPD) : All routing protocols including RIP, OSPF, IS-IS, BGP, PIM, IPv6 counterparts, and so on.
  • User interfaces: Console, Telnet, SSH, J-Web, NetConf.
  • Filesystem interfaces: FTP/SCP.
  • Syslogd: Logging subsystem on the control plane, different than what is on the data plane. This generates the OS and application logs on the control plane.
  • Networking services: DNS, DHCP, NTP, ICMP, ARP/ND, SNMP.
  • Chassisd: Controls the hardware operations of the data plane and interfaces with the components to ensure they are active and operating properly.
  • JSRPD: This is the high availability daemon that runs the HA functionality between two SRX chassis in an HA cluster.

2.2 Data Plane Logging

Data plane logs are primarily those generated by components that process traffic on the data plane. These include the firewall logs (RT_LOG, which stands for Real-Time Log because it is not stored on the data plane) from the flowd process, IPS logs, UTM logs, and logs from other security components like Screens. Data plane logging is off by default and must be configured. Typically, it is recommended that you send logs off the SRX to a syslog host due to the large volume of logs that can be generated from the data plane, particularly on high-end SRX platforms like the 5800. In fact, it can take an entire infrastructure of syslog servers to handle the large volume of syslog messages that the high-end SRX can generate per second. For this reason, there are two different mechanisms that we can use to log messages to the control plane, as discussed in the next section.

Services on the data plane:

  • Intrusion Detection and Prevention Daemon (IDPD)
  • IKED
  • PKID

2.2.1 Event Mode 

Event mode  – control plane log processing – used on low end devices. Optionally even rate can be specified. Once event mode is enabled under “security” then the logging to local file can configure under “system syslog” as above at section 1.2.  You also can configure that security traffic logs are handled through the eventd process and sent with system logs though control panel Routing Engine.

admin@fw-1> show configuration security
log {
    mode event;
    event-rate 1000;
    format sd-syslog;
    source-address 10.9.8.20;
    stream securitylog {
        format sd-syslog;
        category all;
        host {
            10.9.8.52;
        }
    }
    stream LogCollector {
        host {
            10.9.20.17;
        }
    }
    stream TO-10.9.20.33 {
        format sd-syslog;
        category all;
        host {
            10.9.20.33;
        }
    }
}


2.2.2 Stream Mode

Stream mode – data plane logging – Normally used on high end SRX devcies but can be configured on any SRX devices. Under security the syslog parameters can be specified, e.g. syslog server, syslog format, facility.

Note: SRX can only log to the control plane (Event mode) or log out the data plane (Stream mode) at one time

Security logs such as traffic and IDP logs are able to be streamed through the traffic interface ports to a remote syslog server. SRX devices do not send streamed session logs to the Routing Engine (RE). Because system logging is performed on the RE, session or traffic logs cannot be written to the RE file system. Therefore, all traffic logging must be sent to a remote syslog server. Because fxp0 belongs to the RE, the remote syslog server must be reachable by an interface on an IOC. Traffic logging cannot be sent out through fxp0.

When the logging mode is set to stream, security traffic logs generated in the data plane are streamed out a revenue traffic port directly to a remote server. That also means your local log file will stop logging. Match condition configuration in System -> Syslog part does not work in Stream mode.  Its as per design, the Routing engine is the one which puts the match condition and filters the log,
since when we use stream mode the traffic is streamed out of the data plane itself and doesn’t reach the RE the match condition dose not work when using stream mode and only works in event mode.

Basically, only thing works at System – Syslog section are those generated from control plane.

admin@fw-twn1-1> show configuration security
log {
    cache;
    mode stream;
    format sd-syslog;
    source-address 10.2.2.13;
    stream TO-10-0-0-4 {
        format sd-syslog;
        category all;
        host {
            10.0.00.4;
        }
    }
    stream TO-10.4.20.33 {
        format sd-syslog;
        category all;
        host {
            10.4.20.33;
        }
    }
    inactive: traceoptions {
        file jtac;
        flag all;
    }
}

Please keep one thing in mind. Maximum stream destination is three. If you are configuration more than three destination, you will get following error messages from CLI. If you are using JunOS Space, it wont allow you add more than three destination either.

admin@fw-1> commit and-quit
[edit security log]
  ‘stream’
    number of elements exceeds limit of 3
error: commit failed: (number of elements exceeds limit)
Control plane pushing configuration to data plane


admin@fw-srx1> show security log
Security logging is disabled

“show security log” will show you something about audit log but not policy logging after enabled cache in the security log section, else SRX will show you Security Log disabled.

After you enabled cache under security -> log configuration, as shown at the configuration of section 2.2.2, you will get output like below once you use command show security log:

admin@fw-1> show security log
Event time               Message
2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User ‘root’, command ‘xml-mode netconf need-trailer ‘
2015-10-02 09:15:04 UTC  UI_LOGOUT_EVENT: User ‘root’ logout
2015-10-02 09:15:04 UTC  UI_LOGIN_EVENT: User ‘root’ login, class ‘super-user’ [55330], ssh-connection ‘10.4.20.21 7804 10.2.1.14 59097’, client-mode ‘cli’
2015-10-02 09:15:04 UTC  UI_CMDLINE_READ_LINE: User ‘root’, command ‘xml-mode netconf need-trailer ‘


Reference:

By Jon

Leave a Reply