During our regular maintenance, after rebooted one SRX345, and found it stuck at db mode, which is debug mode.

After a short and quick analysis, I found Juniper JunOS devices may get stuck in the boot process or fail to boot the OS, in rare cases, after a sudden power loss or ungraceful power shut down. Juniper  routers, switches and firewalls  can experience file system corruption, which prevents the device from recovering to a functional state. It is recommended that customers minimize their log configurations to prevent excessive read/writes to the file system, which reduces the stress on storage media and reduces the potential occurrence of this issue. Moreover, if abrupt power failures are transient for a very short period of time, the availability of an UPS can also prevent the device from experiencing a sudden power loss.

You do not have to worry about damaging hardware in these situations, as the hardware cannot tell the difference between a graceful shutdown and pulling the power cord. The potential for damage is with the file system structure. It is possible for data to be corrupted, when the computer’s power is interrupted with the operating system running. The data could be in the nodes, which could result in files being lost or file contents being corrupted.

Although rare, this issue more likely occurs on platforms that use a UNIX/BSD-based operating system, such as Junos, to access the flash-based storage media.

“Although rare, file system damage can occur with an abrupt power off, which may cause problems on the next boot. Use the request system halt or request system reboot command to gracefully shut down or reboot the OS. Once the OS is halted, it is safe to remove power.”  – from O’Reilly Media’s JUNOS Enterprise Switching book.

There are a couple of KB discussing the fix. KB29811 is using a USB to copy a snapshot from healthy device to faulty device. KB20046  suggest to press space to go to u-boot prompt and enter some commands to fix issue.



db> help
    DDB Quick Help  
  -------------------  
Type 'c' to continue, 'reset' or 'panic' to restart. 

print       p           examine     x           search      set         write       
w           delete      d           break       dwatch      watch       dhwatch     
hwatch      step        s           continue    c           until       next        
match       trace       alltrace    where       bt          call        show        
ps          gdb         reset       kill        watchdog    thread      panic       
ddbdumpsys  dumpsys     
db> reset

The first reset brought me into RAM mode.

SPI stage 1 bootloader (Build time: May  3 2016 - 23:48:30)
early_board_init: Board type: SRX_345

U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:48:31)

SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10fc00000, size: 0x400000
DRAM: 4 GiB
Clearing DRAM...... done
Using default environment

SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
Found valid SPI bootloader at offset: 0x90000, size: 1481840 bytes


U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:50:19)

Using DRAM size from environment: 4096 MBytes
checkboard siege 
SATA0: not available
SATA1: not available
SATA BIST STATUS = 0x0
SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10f000000, size: 0x1000000
DRAM: 4 GiB
Clearing DRAM...... done
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
PCIe: Port 0 link active, 1 lanes, speed gen2
PCIe: Link timeout on port 1, probably the slot is empty
PCIe: Port 2 not in PCIe mode, skipping
Net:   octrgmii0
octeon_fdt_broadcom_config: Unknown broadcom phy for octrgmii0
Interface 4 has 1 ports (AGL)
Type the command 'usb start' to scan for USB storage devices.

Boot Media: eUSB usb 
Found TPM SLB9660 TT 1.2 by Infineon
TPM initialized
Hit any key to stop autoboot:  0 
Octeon srx_345_ram#  


Octeon srx_345_ram# help     
?           - alias for 'help'
base        - print or set address offset
base64      - print or set address offset
boot        - boot default, i.e., run 'bootcmd'
bootd       - boot default, i.e., run 'bootcmd'
bootelf     - Boot from an ELF image in memory
bootloader  - bootloader - upgrade u-boot
              bootloader - upgrade loader
              bootloader - upgrade ushell
              bootloader - check u-boot
              bootloader - check loader
bootm       - boot application image from memory
bootoct     - Boot from an Octeon Executive ELF image in memory
bootoctelf  - Boot a generic ELF image in memory. NOTE: This command does not
              support simple executive applications, use bootoct for those.
bootoctlinux- Boot from a linux ELF image in memory
bootp       - boot image via network using BOOTP/TFTP protocol
bootvx      - Boot vxWorks from an ELF image
cdp         - Perform CDP network configuration
cmp         - memory compare
cmp64       - memory compare
cp          - memory copy
cp64        - memory copy
cpld        - cpld     - peek/poke CPLD
crc32       - checksum calculation
dhcp        - boot image via network using DHCP/TFTP protocol
dns         - lookup the IP of a hostname
echo        - echo args to console
eeprom      - EEPROM sub-system
env         - environment handling commands
fanspd      - fan speed - set fan speed
fatinfo     - print information about filesystem
fatload     - load binary file from a dos filesystem
fatls       - list files in a directory (default /)
fatwrite    - write file into a dos filesystem
fdt         - flattened device tree utility commands
flush_dcache- Flushes and invalidates the data cache
flush_l2c   - Flushes the L2 cache
freeprint   - Print list of free bootmem blocks
go          - start application at address 'addr'
gpio        - input/set/clear/toggle gpio pins
help        - print command description/usage
i2c         - I2C sub-system
id_eeprom   - id_eeprom     - peek/poke EEPROM
inv_icache  - Invalidates the instruction cache
jump_mdk_mem- jump_mdk_mem - jump to mdk entry
loadb       - load binary file over serial line (kermit mode)
loads       - load S-Record file over serial line
loady       - load binary file over serial line (ymodem mode)
loop        - infinite loop on address range
loop64      - infinite loop on address range
md          - memory display
md5sum      - compute MD5 message digest
md64        - memory display
mdio        - MDIO utility commands
mdkinit     - mdkinit      - start MDK
meminfo     - display memory information
mii         - MII utility commands
mm          - memory modify (auto-incrementing address)
mm64        - memory modify (auto-incrementing address)
mw          - memory write (fill)
mw64        - memory write (fill)
namedalloc  - Allocate a named bootmem block
namedfree   - Free a named bootmem block
namedprint  - Print list of named bootmem blocks
nfs         - boot image via network using NFS protocol
nm          - memory modify (constant address)
nm64        - memory modify (constant address)
octreginfo  - print register information
octwd       - Starts the OCTEON watchdog
pca953x     - pca953x gpio access
pci         - list and access PCI Configuration Space
ping        - send ICMP ECHO_REQUEST to network host
printenv    - print environment variables
qlm         - Octeon QLM debug function (dangerous - remove from final product)
read64      - read 64 bit word from 64 bit address (deprecated)
read64b     - read 8 bit word from 64 bit address (deprecated)
read64l     - read 32 bit word from 64 bit address (deprecated)
read64s     - read 16 bit word from 64 bit address (deprecated)
reset       - Perform RESET of the CPU
run         - run commands in an environment variable
saveenv     - save environment variables to persistent storage
saves       - save S-Record file over serial line
setenv      - set environment variables
sf          - SPI flash sub-system
sha1sum     - compute SHA1 message digest
sleep       - delay execution for some time
sspi        - SPI utility command
test_mdk_mem- test_mdk_mem - start mdk at address 'addr'
tftp        - alias for tftpboot command (deprecated)
tftpboot    - boot image via network using TFTP protocol
tftpput     - TFTP put command, for uploading files to a server
tftpsrv     - act as a TFTP server and boot the first received file
tlv_eeprom  - EEPROM data parsing
tpm         - Issue a TPM command
tpm_test    - tpm_test - Test TPM

usb         - USB sub-system
usbboot     - boot from USB device
version     - print monitor, compiler and linker version
write64     - write 64 bit word to 64 bit address (deprecated)
write64b    - write 8 bit word to 64 bit address (deprecated)
write64l    - write 32 bit word to 64 bit address (deprecated)
write64s    - write 16 bit word to 64 bit address (deprecated)
Octeon srx_345_ram# 

I tried a couple of time reset , reboot, and finally system load backup image.

loader> reboot
Resetting...

SPI stage 1 bootloader (Build time: May  3 2016 - 23:48:30)
early_board_init: Board type: SRX_345

U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:48:31)

SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10fc00000, size: 0x400000
DRAM: 4 GiB
Clearing DRAM...... done
Using default environment

SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
Found valid SPI bootloader at offset: 0x90000, size: 1481840 bytes


U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:50:19)

Using DRAM size from environment: 4096 MBytes
checkboard siege 
SATA0: not available
SATA1: not available
SATA BIST STATUS = 0x0
SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10f000000, size: 0x1000000
DRAM: 4 GiB
Clearing DRAM...... done
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
PCIe: Port 0 link active, 1 lanes, speed gen2
PCIe: Link timeout on port 1, probably the slot is empty
PCIe: Port 2 not in PCIe mode, skipping
Net:   octrgmii0
octeon_fdt_broadcom_config: Unknown broadcom phy for octrgmii0
Interface 4 has 1 ports (AGL)
Type the command 'usb start' to scan for USB storage devices.

Boot Media: eUSB usb 
Found TPM SLB9660 TT 1.2 by Infineon
TPM initialized
Hit any key to stop autoboot:  0 
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
SF: 1048576 bytes @ 0x200000 Read: OK
## Starting application at 0x8f0000a0 ...
Consoles: U-Boot console  
Found compatible API, ver. 3.1
USB1:   
Starting the controller
USB XHCI 1.00
scanning bus 1 for devices... 2 USB Device(s) found
USB0:   
Starting the controller
USB XHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.8
([email protected], Tue Feb 10 00:32:30 PST 2015)
Memory: 4096MB
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
[2]Booting from eUSB slice 1
Loading /boot/defaults/loader.conf 
/kernel data=0xb7fc40+0x1515a4 syms=[0x4+0x9dd10+0x4+0xeb2b7]


Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...               
Kernel entry at 0x801000c0 ...
init regular console
Primary ICache: Sets 16 Size 128 Asso 39
Primary DCache: Sets 8 Size 128 Asso 32
Secondary DCache: Sets 1024 Size 128 Asso 4
CIU_FUSE 0xf/0xf
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Running in PARTITIONED TLB MODE
Copyright (c) 1996-2016, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
JUNOS 15.1X49-D45 #0: 2016-04-25 07:29:58 UTC
    [email protected]:/volume/build/junos/15.1/service/15.1X49-D45/obj/octeon/junos/bsd/kernels/JSRXNLE/kernel
can't re-use a leaf (debug)!
JUNOS 15.1X49-D45 #0: 2016-04-25 07:29:58 UTC
    [email protected]:/volume/build/junos/15.1/service/15.1X49-D45/obj/octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory  = 4294967296 (4194304K bytes)
avail memory = 2621882368 (2500MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
Security policy loaded: Junos MAC/veriexec (mac_veriexec)
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
MAC/veriexec fingerprint module loaded: SHA256
MAC/veriexec fingerprint module loaded: SHA1
netisr_init: !debug_mpsafenet, forcing maxthreads from 4 to 1
cpu0 on motherboard
: CAVIUM's OCTEON 70XX/71XX CPU Rev. 0.2 with no FPU implemented
        L1 Cache: I size 78kb(128 line), D size 32kb(128 line), thirty two way.
        L2 Cache: Size 512kb, 4 way
obio0 on motherboard
uart0: <Octeon-16550 channel 0> on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
set clock 0x58
xhci0: <Cavium Octeon 7xxx xHCI Host Driver> on obio0
usb0: <USB bus for xHCI Controller> on xhci0
usb0: USB revision 3.0
uhub0: vendor 0x0000 XHCI root hub, class 9/0, rev 3.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
xhci1: <Cavium Octeon 7xxx xHCI Host Driver> on obio0
usb1: <USB bus for xHCI Controller> on xhci1
usb1: USB revision 3.0
uhub1: vendor 0x0000 XHCI root hub, class 9/0, rev 3.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
cpld0 on obio0
pcib0: <Cavium on-chip PCIe HOST bridge> on obio0
Disabling Octeon big bar support
pcib0: Initialized controller
pci0: <PCI bus> on pcib0
pci0: <network, ethernet> at device 0.0 (no driver attached)
pci0: <network, ethernet> at device 0.1 (no driver attached)
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <Macronix MX25L64 - 8MB> on obio0
cfi1: <Macronix MX25L64 - 8MB> on obio0
octagl0: <Octeon AGL> on obio0
umass0: ATP Electronics ATP CG eUSB, rev 2.00/11.00, addr 2
miibus0: <MII bus> on octagl0
brgphy0: <BCM54616S 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Timecounter "mips" frequency 1600000000 Hz quality 0
Registered AMT tunnel Encap with UDP Tunnel!
 Loading Redundant LT driver
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
Kernel thread "wkupdaemon" (pid 48) exited prematurely.
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ATP ATP CG eUSB 1100> Fixed Direct Access SCSI-4 device 
da0: 40.000MB/s transfers
da0: 7672MB (15712256 512 byte sectors: 255H 63S/T 978C)
Trying to mount root from ufs:/dev/da0s1a
MFSINIT: Initialising MFSROOT 
Process-1 beginning MFSROOT initialization...
Creating MFSROOT...
/dev/md0: 20.0MB (40956 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 5.00MB, 320 blks, 640 inodes.
super-block backups (for fsck -b #) at:
 32, 10272, 20512, 30752
Populating MFSROOT...
Creating symlinks...
Setting up mounts...
Continuing boot from MFSROOT...
Attaching /cf/packages/junos via /dev/mdctl...
Mounted junos package on /dev/md1...
O
WARNING: R/W mount of /cf/var denied.  Filesystem is not clean - run fsck
mount: /dev/bo0s3f : Operation not permitted
chflags: /var/packages/*: No such file or directory
umount: /dev/bo0s3f: unknown file system
Automatic reboot in progress...
** /dev/da0s1a (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
163 files, 115537 used, 1148817 free (65 frags, 143594 blocks, 0.0% fragmentation)
mount reload of '/' failed: Operation not supported 

Verified junos signed by PackageProductionEc_2016
Verified jboot signed by PackageProductionEc_2016
Verified junos-15.1X49-D45-domestic signed by PackageProductionEc_2016
Checking integrity of BSD labels:
  s1: Passed
  s2: Passed
  s3: Passed
  s4: Passed
** /dev/bo0s3e
** Last Mounted on /config
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
18 files, 65 used, 94711 free (39 frags, 11834 blocks, 0.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****
** /dev/bo0s3f
** Last Mounted on /cf/var
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=70677 (480 should be 416)
CORRECT? yes

** Phase 2 - Check Pathnames
DIRECTORY CORRUPTED  I=117761  OWNER=0 MODE=40755
SIZE=512 MTIME=Sep 18 14:41 2017 
DIR=?

SALVAGE? yes

MISSING '.'  I=117761  OWNER=0 MODE=40755
SIZE=512 MTIME=Sep 18 14:41 2017 
DIR=?

FIX? yes

MISSING '..'  I=117761  OWNER=0 MODE=40755
SIZE=512 MTIME=Sep 18 14:41 2017 
DIR=/etc

FIX? yes

** Phase 3 - Check Connectivity
UNREF DIR  I=117766  OWNER=0 MODE=40755
SIZE=512 MTIME=Sep 28 06:06 2016 
RECONNECT? yes

DIR I=117766 CONNECTED. PARENT WAS I=117761

UNREF DIR  I=117762  OWNER=0 MODE=40755
SIZE=512 MTIME=Oct 17 21:07 2016 
RECONNECT? yes

DIR I=117762 CONNECTED. PARENT WAS I=117761

** Phase 4 - Check Reference Counts
UNREF FILE  I=47130  OWNER=0 MODE=100644
SIZE=0 MTIME=Sep 18 14:40 2017 
RECONNECT? yes

LINK COUNT DIR I=117761  OWNER=0 MODE=40755
SIZE=512 MTIME=Sep 18 14:41 2017  COUNT 4 SHOULD BE 2
ADJUST? yes

UNREF FILE  I=117765  OWNER=0 MODE=120755
SIZE=22 MTIME=Sep 28 06:06 2016 
RECONNECT? yes

UNREF FILE  I=117767  OWNER=0 MODE=100644
SIZE=1817 MTIME=Aug 21 01:10 2017 
RECONNECT? yes

UNREF FILE  I=117794  OWNER=0 MODE=100644
SIZE=40960 MTIME=Sep 18 14:41 2017 
RECONNECT? yes

LINK COUNT FILE I=117794  OWNER=0 MODE=100644
SIZE=40960 MTIME=Sep 18 14:41 2017  COUNT 2 SHOULD BE 1
ADJUST? yes

UNREF FILE  I=117797  OWNER=0 MODE=100600
SIZE=40960 MTIME=Sep 18 14:41 2017 
RECONNECT? yes

UNREF FILE  I=117800  OWNER=0 MODE=100644
SIZE=1195 MTIME=Sep 18 14:41 2017 
RECONNECT? yes

UNREF FILE  I=117801  OWNER=0 MODE=100644
SIZE=40960 MTIME=Aug 21 01:10 2017 
RECONNECT? yes

UNREF FILE  I=117802  OWNER=0 MODE=100600
SIZE=40960 MTIME=Aug 21 01:10 2017 
RECONNECT? yes

UNREF FILE  I=117803  OWNER=0 MODE=100644
SIZE=1195 MTIME=Aug 21 01:10 2017 
RECONNECT? yes

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes

SUMMARY INFORMATION BAD
SALVAGE? yes

BLK(S) MISSING IN BIT MAPS
SALVAGE? yes

459 files, 277705 used, 831508 free (340 frags, 103896 blocks, 0.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
Checking integrity of licenses:
Checking integrity of configuration:
  rescue.conf.gz: Passed
cd: can't cd to /etc/db/pkg
hw.re.gres_sync_other: 0 -> 1
Loading configuration ...
Time and ticks drifted too much,                        resetting synchronization...
Non-existant dump device /dev/bo0s1b
mgd: commit complete
Setting initial options: .
Starting optional daemons:  usbd.
Doing initial network setup:.
Initial interface configuration:
additional daemons: eventd.
Non-existant dump device /dev/bo0s1b
Additional routing options:kern.module_path: /boot//kernel;/boot/modules -> /boot/modules;/modules/ifpfe_drv;IPsec: Initialized Security Association Processing.
/modules;
k.
Doing additional network setup:.
Starting final network daemons:.
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
Initial rc.mips initialization:.
Local package initialization:.
starting local daemons:hw.re.gres_sync_other: 1 -> 0
set cores for group access
.
kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
Boot media /dev/da0 has dual root support
WARNING: JUNOS versions running on dual partitions are not same
** /dev/da0s2a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 1152347 free (59 frags, 144036 blocks, 0.0% fragmentation)
Wed Oct  4 13:26:20 UTC 2017
OOOOOOOOOOOOOO
fw-prod-1 (ttyu0)

login: john
Password:
Login incorrect
login: Sec3166
Password:

Login incorrect
login: login: john
Password:






--- JUNOS 15.1X49-D45 built 2016-04-25 07:29:58 UTC

***********************************************************************
**                                                                   **
**  WARNING: THIS DEVICE HAS BOOTED FROM THE BACKUP JUNOS IMAGE      **
**                                                                   **
**  It is possible that the primary copy of JUNOS failed to boot up  **
**  properly, and so this device has booted from the backup copy.    **
**                                                                   **
**  Please re-install JUNOS to recover the primary copy in case      **
**  it has been corrupted and if auto-snapshot feature is not        **
**  enabled.                                                         **
**                                                                   **
***********************************************************************

{secondary:node0}
john@fw-prod-1> 
{secondary:node0}
john@fw-prodtrans-mex1-1> request system reboot 
Reboot the system ? [yes,no] (no) yes 

Shutdown NOW!
[pid 10707]

                                                                               
*** FINAL System shutdown message from john@fw-prodtrans-mex1-1 ***          

System going down IMMEDIATELY                                                  

                                                                               
{secondary:node0}
john@fw-prodtrans-mex1-1> OOWaiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 done

syncing disks... Syncing disks, buffers remaining... 10 10 10 10 10 8 8 8 8 8 8 8 4 4 4 4 4 4 4 2 2 2 2 2 2 2 
Final sync complete
Uptime: 6d7h9m4s
Rebooting...
cpu_reset: Stopping other CPUs


SPI stage 1 bootloader (Build time: May  3 2016 - 23:48:30)
early_board_init: Board type: SRX_345

U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:48:31)

SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10fc00000, size: 0x400000
DRAM: 4 GiB
Clearing DRAM...... done
Using default environment

SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
Found valid SPI bootloader at offset: 0x90000, size: 1481840 bytes


U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:50:19)

Using DRAM size from environment: 4096 MBytes
checkboard siege 
SATA0: not available
SATA1: not available
SATA BIST STATUS = 0x0
SRX_345 board revision major:1, minor:7, serial #: CZ2616AF0301
OCTEON CN7130-AAP pass 1.2, Core clock: 1600 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10f000000, size: 0x1000000
DRAM: 4 GiB
Clearing DRAM...... done
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
PCIe: Port 0 link active, 1 lanes, speed gen2
PCIe: Link timeout on port 1, probably the slot is empty
PCIe: Port 2 not in PCIe mode, skipping
Net:   octrgmii0
octeon_fdt_broadcom_config: Unknown broadcom phy for octrgmii0
Interface 4 has 1 ports (AGL)
Type the command 'usb start' to scan for USB storage devices.

Boot Media: eUSB usb 
Found TPM SLB9660 TT 1.2 by Infineon
TPM initialized
Hit any key to stop autoboot:  0 
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
SF: 1048576 bytes @ 0x200000 Read: OK
## Starting application at 0x8f0000a0 ...
Consoles: U-Boot console  
Found compatible API, ver. 3.1
USB1:   
Starting the controller
USB XHCI 1.00
scanning bus 1 for devices... 2 USB Device(s) found
USB0:   
Starting the controller
USB XHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.8
([email protected], Tue Feb 10 00:32:30 PST 2015)
Memory: 4096MB
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
[0]Booting from eUSB slice 2
\
can't load '/kernel'
can't load '/kernel.old'
Press Enter to stop auto bootsequencing and to enter loader prompt.


Type '?' for a list of commands, 'help' for more detailed help.
loader> reboot
Resetting...

SPI stage 1 bootloader (Build time: May  3 2016 - 23:48:30)
early_board_init: Board type: SRX_345

In KB26175, if the following error messages has been displayed  , format install will be the last recovery option to resolve it.

can’t load ‘/kernel’
can’t load ‘/kernel.old’

The backup image is not same as primary. The fix is to install the required image 15.1×49-D50.3 to primary partition and reboot it.

john@fw-prodtrans-1> show version 
node0:
--------------------------------------------------------------------------
Hostname: fw-prodtrans-1
Model: srx345
Junos: 15.1X49-D45
JUNOS Software Release [15.1X49-D45]

node1:
--------------------------------------------------------------------------
Hostname: fw-prodtrans-2
Model: srx345
Junos: 15.1X49-D50.3
JUNOS Software Release [15.1X49-D50.3]

john@fw-prodtrans-1# run show chassis routing-engine node 0  
node0:
--------------------------------------------------------------------------
Routing Engine status:
    Temperature                 35 degrees C / 95 degrees F
    CPU temperature             70 degrees C / 158 degrees F
    Total memory              4096 MB Max  1229 MB used ( 30 percent)
      Control plane memory    2624 MB Max   551 MB used ( 21 percent)
      Data plane memory       1472 MB Max   662 MB used ( 45 percent)
    5 sec CPU utilization:
      User                       6 percent
      Background                 0 percent
      Kernel                     1 percent
      Interrupt                  0 percent
      Idle                      94 percent
    Model                          RE-SRX345
    Serial ID                      CZ2616AF0301
    Start time                     2017-10-04 13:22:28 UTC
    Uptime                         6 days, 7 hours, 5 minutes, 56 seconds
    Last reboot reason             0x800:reboot due to exception
    Load averages:                 1 minute   5 minute  15 minute
                                       0.05       0.08       0.03
{secondary:node0}
john@fw-prodtrans-1> show chassis cluster status 
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring              
    SP  SPU monitoring              SM  Schedule monitoring
    CF  Config Sync monitoring
 
Cluster ID: 9
Node   Priority Status         Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 0
node0  0        secondary      no      no       CF             
node1  100      primary        no      no       None           

Redundancy group: 1 , Failover count: 0
node0  0        secondary      no      yes      CF             
node1  255      primary        no      yes      None   
{secondary:node0}
john@fw-prodtrans-1> show security flow session summary 
node0:
--------------------------------------------------------------------------
Unicast-sessions: 128
Multicast-sessions: 0
Failed-sessions: 0
Sessions-in-use: 133
  Valid sessions: 128
  Pending sessions: 0
  Invalidated sessions: 5
  Sessions in other states: 0
Maximum-sessions: 384000

node1:
--------------------------------------------------------------------------
Unicast-sessions: 123
Multicast-sessions: 0
Failed-sessions: 0
Sessions-in-use: 139
  Valid sessions: 123
  Pending sessions: 0
  Invalidated sessions: 16
  Sessions in other states: 0
Maximum-sessions: 384000
{secondary:node0}
john@fw-prodtrans-1> request system software add /var/tmp/junos-15.1X49-D50.3-domestic.tgz no-copy no-validate reboot         
Oct 10 20:40:53 init: gstatd (PID 2300) exited with status=1 
Oct 10 20:40:53 init: exec_command: /usr/sbin/gstatd (PID 2308) started
Oct 10 20:40:53 init: gstatd (PID 2308) started
Formatting alternate root (/dev/da0s2a)...
/dev/da0s2a: 2518.0MB (5156848 sectors) block size 16384, fragment size 2048
        using 14 cylinder groups of 183.62MB, 11752 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
 32, 376096, 752160, 1128224, 1504288, 1880352, 2256416, 2632480, 3008544,
 3384608, 3760672, 4136736, 4512800, 4888864
Installing package '/altroot/cf/packages/install-tmp/junos-15.1X49-D50.3-domestic' ...
Verified junos-boot-srxsme-15.1X49-D50.3.tgz signed by PackageProductionEc_2016
Verified junos-srxsme-15.1X49-D50.3-domestic signed by PackageProductionEc_2016
Verified junos-boot-srxsme-15.1X49-D50.3.tgz signed by PackageProductionRSA_2016
Verified junos-srxsme-15.1X49-D50.3-domestic signed by PackageProductionRSA_2016
JUNOS 15.1X49-D50.3 will become active at next reboot
cp: cannot overwrite directory /altroot/cf/etc/ssh with non-directory /cf/etc/ssh
Saving state for rollback ...
Rebooting ...
shutdown: [pid 2671]
Shutdown NOW!
                                                                               
*** FINAL System shutdown message from root@fw-prodtrans-1 ***          

System going down IMMEDIATELY       




After successfully installed new image , reboot was smooth to get me into regular login window.

By Jon

Leave a Reply