For SRX1400, SRX3400, SRX3600, SRX5600, and SRX5800 devices, command introduced in Junos OS Release 9.6 and support for reboot as a required parameter added in Junos OS Release 11.2R2. For SRX100, SRX210, SRX220, SRX240, and SRX650 devices, command introduced in Junos OS Release 11.2R2. For SRX5400 devices, the command is introduced in Junos OS Release 12.1X46-D20.
Table of Contents
Symptoms:
Symptom 1: “tar: Archive contains obsolescent base-64 headers”
root@fw-1> request system software add no-copy /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz no-validate
Formatting alternate root (/dev/da0s2a)...
/dev/da0s2a: 627.4MB (1284940 sectors) block size 16384, fragment size 2048
using 4 cylinder groups of 156.86MB, 10039 blks, 20096 inodes.
super-block backups (for fsck -b #) at:
32, 321280, 642528, 963776
Extracting /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz ...
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
gzip: stdin: invalid compressed data--format violated
tar: Child returned status 1
tar: Error exit delayed from previous errors
ERROR: Failed to extract /var/tmp/junos-srxsme-12.1X44-D40.2-domestic.tgz to /altroot/cf/packages/install-tmp/junos-12.1X44-D40.2-domestic
Cause:
This error usually occurs when a file is transferred over to the server in ASCII mode or ftp server software itself issue. To resolve the issue, transfer the file over to the server again; this time in the binary mode. The best practice is to verify file integrity after uploaded the package to the device.
Solution:
using WinSCP
root@fw-1% ftp 10.9.1.109
Connected to 10.9.1.109.
220 Welcome to Quick 'n Easy FTP Server
Name (10.9.1.109:root): test
331 Password required for test
Password:
230 User successfully logged in.
Remote system type is UNIX.
ftp> bin
200 Type set to BINARY
ftp> get junos-srxsme-12.1X44-D40.2-domestic.tgz
local: junos-srxsme-12.1X44-D40.2-domestic.tgz remote: junos-srxsme-12.1X44-D40.2-domestic.tgz
200 Port command successful.
150 Opening BINARY mode data connection for file transfer.
29% |************** | 42615 KB 07:26 ETA^ | 1964 KB 19:34 ETAC
Symptom 2: ISSU in progress error
I am having a hiccup when upgrading JunOS on SRX1400. ISSU upgrade failed because of not enough space on CF folder. Here is the output of failed upgrade process:
root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy
Chassis ISSU Started
node0:
--------------------------------------------------------------------------
Chassis ISSU Started
ISSU: Validating Image
Initiating in-service-upgrade
node0:
--------------------------------------------------------------------------
Initiating in-service-upgrade
Checking compatibility with configuration
Initializing...
Verified manifest signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Using /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz
Checking junos requirements on /
Available space: 289258 require: 24128
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
Verified manifest signed by PackageProduction_12_1_0
Hardware Database regeneration succeeded
Validating against /config/juniper.conf.gz
Usage: license-check -f "<features>" -m -p -q -M -u -U -V
-V verify if release based licenses are present
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
Validating against /config/rescue.conf.gz
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
failover all RG 1+ groups to node 1
Initiated manual failover for all redundancy-groups to node1
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: Preparing Backup RE
Pushing bundle to node0
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 289258 require: 297584
WARNING: The /cf filesystem is low on free disk space.
WARNING: This package requires 297584k free, but there
WARNING: is only 289258k available.
WARNING: This installation attempt will be aborted.
ERROR: junos-12.1X46-D25.7-domestic fails requirements check
Installation failed for package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz'
error: Failed to install image on secondary node (error-code: 3.1)
error: ISSU Aborted! Backup node maybe in inconsistent state, Please restore backup node
ISSU aborted. But, both nodes are in ISSU window.
Please do the following:
1. Rollback the node with the newer image using rollback command
Note: use the ‘node’ option in the rollback command
otherwise, images on both nodes will be rolled back
2. Make sure that both nodes (will) have the same image
3. Ensure the node with older image is primary for all RGs
4. Abort ISSU on both nodes
5. Reboot the rolled back node
Solutions:
1. Clear system to get enough space for upgrading process
{secondary:node0}
root@fw-1-1> request system software delete-backup
Delete backup system software package [yes,no] (no) yes
{secondary:node0}
root@fw-1-1> show system storage node 0
node0:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 891M 537M 282M 66% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 523M 523M 0B 100% /junos
/cf 891M 537M 282M 66% /junos/cf
devfs 1.0K 1.0K 0B 100% /junos/dev/
procfs 4.0K 4.0K 0B 100% /proc
/dev/ad0s1e 99M 102K 91M 0% /config
/dev/ad2s1f 12G 461M 10G 4% /var
/dev/md1 1006M 1.0M 924M 0% /mfs
/var/jail 12G 461M 10G 4% /jail/var
/var/log 12G 461M 10G 4% /jail/var/log
devfs 1.0K 1.0K 0B 100% /jail/dev
{secondary:node0}
root@fw-1-1> file delete /cf/packages/?
Possible completions:
<[Enter]> Execute this command
<path> Path to delete
/cf/packages/junos Size: 280224023, Last changed: Aug 28 15:07:56
/cf/packages/junos-12.1X44-D40.2-domestic Size: 280224023, Last changed: Aug 28 15:07:56
/cf/packages/junos-12.1X44-D40.2-domestic.certs Size: 7153, Last changed: Aug 28 10:15:58
/cf/packages/junos-12.1X44-D40.2-domestic.sha1 Size: 41, Last changed: Aug 28 15:08:54
/cf/packages/junos-12.1X44-D40.2-domestic.sig Size: 525, Last changed: Aug 28 15:09:27
/cf/packages/junos-srx1k3k-12.1X44-D25.5-domestic Size: 279916118, Last changed: May 13 2014
/cf/packages/junos.old Size: 279916118, Last changed: May 13 2014
{secondary:node0}
root@fw-1-1> file delete /cf/packages/junos-srx1k3k-12.1X44-D25.5-domestic
{secondary:node0}
root@fw-1-1> show system storage detail
node0:
--------------------------------------------------------------------------
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/ad0s1a 912232 276540 562714 33% /
devfs 1 1 0 100% /dev
/dev/md0 535230 535230 0 100% /junos
/cf 912232 276540 562714 33% /junos/cf
devfs 1 1 0 100% /junos/dev/
procfs 4 4 0 100% /proc
/dev/ad0s1e 101202 102 93004 0% /config
/dev/ad2s1f 12432412 472166 10965654 4% /var
/dev/md1 1029676 1046 946256 0% /mfs
/var/jail 12432412 472166 10965654 4% /jail/var
/var/log 12432412 472166 10965654 4% /jail/var/log
devfs 1 1 0 100% /jail/dev
node1:
--------------------------------------------------------------------------
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/ad0s1a 912232 276540 562714 33% /
devfs 1 1 0 100% /dev
/dev/md0 535230 535230 0 100% /junos
/cf 912232 276540 562714 33% /junos/cf
devfs 1 1 0 100% /junos/dev/
procfs 4 4 0 100% /proc
/dev/ad0s1e 101202 102 93004 0% /config
/dev/ad2s1f 12432412 830654 10607166 7% /var
/dev/md1 1029676 11190 936112 1% /mfs
/var/jail 12432412 830654 10607166 7% /jail/var
/var/log 12432412 830654 10607166 7% /jail/var/log
devfs 1 1 0 100% /jail/dev
SW1#
2. Clear ISSU in progress error
After got enough space on both cluster members, I tried another ISSU upgrade but there is another warning message:
root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy
warning: ISSU in progress
///Then another one for abort command
root@fw-1-2> request system software abort in-service-upgrade
error: command is not valid on the srx1400
///Finally got the right command for SRX1400
root@fw-1-2> request chassis cluster in-service-upgrade abort
Exiting in-service-upgrade window
Chassis ISSU Aborted
3. Successfully upgraded SRX1400 to 12.1X46-D25.7
root@fw-1-2> request system software in-service-upgrade /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz reboot no-copy
Chassis ISSU Started
Chassis ISSU Started
ISSU: Validating Image
Initiating in-service-upgrade
Initiating in-service-upgrade
Checking compatibility with configuration
Initializing...
Verified manifest signed by PackageProduction_12_1_0
Verified junos-12.1X44-D40.2-domestic signed by PackageProduction_12_1_0
Using /var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz
Checking junos requirements on /
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
Verified manifest signed by PackageProduction_12_1_0
Hardware Database regeneration succeeded
Validating against /config/juniper.conf.gz
Usage: license-check -f "<features>" -m -p -q -M -u -U -V
-V verify if release based licenses are present
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
Validating against /config/rescue.conf.gz
FIPS daemon: warning: request security internal-security-association refresh to take effect
FIPS daemon:
mgd: commit complete
Validation succeeded
failover all RG 1+ groups to node 1
Initiated manual failover for all redundancy-groups to node1
Redundancy-groups-0 will not failover and the primaryship remains unchanged.
ISSU: Preparing Backup RE
Pushing bundle to node0
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
JUNOS 12.1X46-D25.7 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING: Use the 'request system reboot' command
WARNING: when software installation is complete
Saving state for rollback ...
Finished upgrading secondary node node0
Rebooting Secondary Node
Shutdown NOW!
[pid 90847]
ISSU: Backup RE Prepare Done
Waiting for node0 to reboot.
node0 booted up.
Waiting for node0 to become secondary
node0 became secondary.
Waiting for node0 to be ready for failover
ISSU: Preparing Daemons
Secondary node0 ready for failover.
Installing package '/var/tmp/junos-srx1k3k-12.1X46-D25.7-domestic.tgz' ...
Verified junos-boot-srx1k3k-12.1X46-D25.7.tgz signed by PackageProduction_12_1_0
Verified junos-srx1k3k-12.1X46-D25.7-domestic signed by PackageProduction_12_1_0
Available space: 562714 require: 297584
Saving boot file package in /var/sw/pkg/junos-boot-srx1k3k-12.1X46-D25.7.tgz
JUNOS 12.1X46-D25.7 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING: Use the 'request system reboot' command
WARNING: when software installation is complete
Saving state for rollback ...
Failing over all redundancy-groups to node0
ISSU: Preparing for Switchover
Initiated failover for all the redundancy groups to node0
Waiting for node0 take over all redundancy groups
{secondary:node1}
john@fw-1-2> show chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Online SRX1k Dual Wide NPC+SPC Support Card
PIC 0 Online SPU Cp-Flow
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
node1:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Present SRX1k Dual Wide NPC+SPC Support Card
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
{secondary:node1}
john@fw-1-2> show chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Online SRX1k Dual Wide NPC+SPC Support Card
PIC 0 Online SPU Cp-Flow
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
node1:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Online SRX1k Dual Wide NPC+SPC Support Card
PIC 0 Offline
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
john@fw-1-2> show chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Online SRX1k Dual Wide NPC+SPC Support Card
PIC 0 Online SPU Cp-Flow
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
node1:
--------------------------------------------------------------------------
Slot 0 Online SRX1k 10GE SYSIO
PIC 0 Online 6x 1GE RJ45 3x 1GE SFP 3x 10GE SFP+
Slot 1 Online SRX1k Dual Wide NPC+SPC Support Card
PIC 0 Online SPU Cp-Flow
Slot 3 Online BUILTIN NPC
PIC 0 Online NPC PIC
Symptom 3: ISSU not allowed
ohn@fw-com1-1> request system software in-service-upgrade /var/tmp/junos-srxsme-15.1X49-D50.3-domestic.tgz no-sync
WARNING: Not enabled dual root partition on secondary node
ISSU not allowed
Solution:
Using regular upgrade method.
Fond more information here.