Juniper vMX (Virtual MX) Router in VMWare ESXi 5.5

The VMX is Juniper’s latest product in it’s virtual products portfolio. Official page is https://www.juniper.net/us/en/products-services/routing/mx-series/vmx/.
Update: Juniper Virtual MX (vMX) 14.2R5.3 Phase 2 (RE and MPC on different VMs).

At the time of this article, VMX is not released yet, but it can be obtained for beta testing through sales channel so you can request to IMG image which is qcow2 QEMU file format for usage with KVM hypervisor.

The virtual MX deployment had two phases:
Phase 1 based on Junos 14.1 having the Routing Engine and the virtual PFE in the same virtual machine (with a note that releases above 14.1R4 need a tweak to enable the local PFE instead of using a separate VM)
Phase 2 based on Junos 14.2 splitting the RE and PFE in different virtual machines and using an IP connection between them for communication.

I’m going to provide a short tutorial for deploying Phase 1 vMX running 14.1R4.9 with the tweak to enable the local virtual PFE (in the same VM).

To install the virtual MX router under VMWare ESXi hypervisor, we need to convert the qcow2 img vmx image, that it is shipped at this time, to vmdk disk format (vmware specific).

Converting vMX qcow2 / img disk to VMWare vmdk format

Copy the IMG vmx disk file on a QEMU installed system and run the following command to convert to vmdk:


# qemu-img convert -f qcow2 -O vmdk jinstall-vmx-14.1R4.9-domestic.img VMX-P6-14.1R4.9.vmdk  

Setting the VMWare ESXi host to import the newly created VMDK disk

Once the VMDK disk is created, it needs to be copied to the ESXi host. Since this will be virtual machine disk that will get imported, we need to set up a special directory to reflect the VM name. In my case, I use it as P6 router, so I will make a /vmfs/volumes/datastore1/VMX-P6-14.1R4 directory for this vMX and copy the VMX-P6-14.1R4.9.vmdk disk inside it.

Creating the vMX virtual machine in vsphere client

In this step, it is required to create a new virtual machine and follow the next steps:
– Use a “Custom” configuration.
– Provide the name of the VM: “VMX-P6-14.1R4″ in my case.
– Select resource pool or use the default.
– Select the destination datastore for VM files.
– Select “Virtual Machine Version: 8″
– Select “Other” and “Other (64-bit)” operating system.
– Select at least 2 CPUs for the VM (one virtual socket and 2 cores per virtual socket in my case).
– Select 1 GB RAM memory.
– Add 4 NICs with E1000 adapter.
– Select “LSA Logic SAS” adapter.
– Chose “Use an existing virtual disk”.
– Browse the ESXi localstore holding the above VMDK disk file.

Note: At this stage it is important to know that vMX has 4 interfaces: em0, em1, em2 and em3. VMX will map em2 to ge-0/0/0, em3 to ge-0/0/1 and so on until ge-0/0/9 so the 3rd interface assigned to the virtual machine (em2) should be connected to the ESXi vSwitch that connects to the management LAN to provide access.

First steps to set up the vMX running 14.1R4.9 Junos

Using the vSphere client, power on and open the vMX console and wait for it to provide the login prompt. Login as “root” and you will be logged to the BSD shell.
Use “cli” command to go into Junos CLI and “conf” to start the configuration.
vmx-post-install-first-steps

vmx-post-install-first-steps-2

Issuing the “run show chassis fpc pic-status” from Junos configuration mode, you will see that the Virtual PFE is in “Present” state, not “Online” as it should. After a few minutes it will go into “Offline” state.

This is because vMX based on 14.1R4+ will try to connect to a remote PFE (different virtual machine) using IP subnet 172.16.0.1/16 on em1 interface used for RE and distributed PFE communication, but I do not have a distributed PFE so I want to use local PFE. To enable it, add vm_local_rpio=”1″ line to /boot/loader.conf and save the file. This will take effect after next reboot.

While still in configuration mode, we need to enable ge-0/0/0 interface to hook into traffic coming on the 3rd interface (em2). To do so, ge-0/0/0 interface needs static mac assignment as the mac of em2 interface. Also ge-0/0/0 will need an IP address from the management LAN.

The next step is to set a root password because without it, Junos will not allow any commits.

Now reboot vMX with “run request system reboot” and confirmation.
After reboot, the virtual PFE will be “Online” and it will bring up ge-0/0/0 to ge-0/0/9 interfaces and the vMX will be accessible via ssh:


$ ssh 10.1.1.156
The authenticity of host '10.1.1.156 (10.1.1.156)' can't be established.
ECDSA key fingerprint is 6d:b1:01:0b:af:0f:ea:26:ba:6b:a0:66:ad:80:ce:8a.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.156' (ECDSA) to the list of known hosts.
Password:
Password:
--- JUNOS 14.1R4.9 built 2015-02-01 01:53:57 UTC
root@% 
root@% cli
root> show chassis fpc pic-status 
Slot 0   Online       Virtual FPC
  PIC 0  Online       Virtual 10x1GE PIC

root> show interfaces ge* terse 
Interface               Admin Link Proto    Local                 Remote
ge-0/0/0                up    up
ge-0/0/0.0              up    up   inet     10.1.1.156/24   
                                   multiservice
ge-0/0/1                up    up
ge-0/0/2                up    up
ge-0/0/3                up    up
ge-0/0/4                up    up
ge-0/0/5                up    up
ge-0/0/6                up    up
ge-0/0/7                up    up
ge-0/0/8                up    up
ge-0/0/9                up    up

root> show interfaces em* terse 
Interface               Admin Link Proto    Local                 Remote
em1                     up    up
em1.0                   up    up   inet     172.16.0.1/16   
                                   inet6    fe80::20c:29ff:fe2a:c1b7/64
em2                     up    up
em3                     up    up

root> show interfaces fxp* terse 
Interface               Admin Link Proto    Local                 Remote
fxp0                    up    up

Let’s discuss the above output:
– Virtual FPC0 (the local virtual PFE is online).
– 1 there are 4 interfaces assigned to the vMX virtual machine and they are mapped in the following manner:
1st NIC – fxp0 – management interface hooked directly into the Routing Engine
2nd NIC – em0
3rd NIC – em1 – Used for RE to distributed PFE communication
4th NIC – em2 – ge-0/0/0

13th NIC – em11 – ge-0/0/9

I used ge-0/0/0 here for the management lan and not fxp0 interface to make the interface mapping more visible, but fxp0 interface should be used for management, so let’s fix that:


[edit]
root> configure 
Entering configuration mode

[edit]
root# delete interfaces ge-0/0/0 

[edit]
root# set interfaces fxp0.0 family inet address 10.1.1.156/24 

[edit]
root# set system host-name P6 

[edit]
root# commit 
commit complete

[edit]
root@P6# 

At this point, virtual MX is set up, has a hostname P6, uses 1st interface as fxp0 as an RE interface, 2nd interface as em0 (not clear to me atm what it is used for), 3rd interface as em1 used for RE / PFE communication, 4th -to 13th interfaces as em2-em11 hooked into the PFE mapped as ge-0/0/0 to ge-0/0/9 interfaces once these ge* interfaces are configured to used the mac addresses of the corresponding em2+ interfaces.

vMX is intended to have feature parity with the Juniper’s physical MX 3D router and I have tested to work:
– BGP RR, iBGP, eBGP, multipath, Add-path.
– LDP, RSVP, IGP TE, RSVP-TE, CSPF, VPLS, L3VPN, L2VPN, EVPN.
– virtual bridges.
Flowspec I havent tested.


[Admin Edit]: I have created following forum for vMX related discussions: https://forum.ivorde.com/juniper-virtual-mx-vmx-f146.html

Posted in Juniper, Juniper Virtual SRX / MX and tagged , .

52 Comments

  1. Hi

    DO you know how connect MPC2 MODULE TO ( vMX14.2-MPC2-vm.vmdk) TO RE0 VIRTUAL MACHINE ?

    the new VMX 14.2 has been released as well

  2. Hi, did you have any 100% CPU usage issue? I can’t get rid of it. It works, but vMX constantly consumes whole CPU power.. Any ideas how to avoid it?

    • Check your RE CPU utilization (show chassis routing-engine) and check your interfaces counters (> monitor interface traffic). Using OSPF/ISIS/BGP ?

      • There is “riot” process that consumes all the CPU power.. I have almost nothing in my configuration, no protocols, no services, no addresses etc. The same is when I use only one machine and enable not mentioned officialy vm_local_rpio=”1″ parameter in loader.conf.

  3. is there any difference between the phase 1 and phase II in terms of features supported,
    do u have any idea ??

    • Unfortunately I dont have a comprehensive list of features roadmap for VMX. Only thing I can see so far is the EVPN supporting the default gateway extended community starting with 15.2 and also per vlan label allocation starting from 14.1R3 instead of per EVI.

  4. Hi,
    is it possible to have two REs in vMX to test some of the redundancy options in juniper ? if yes pls let me know how to configure to have two REs with one MPC
    [EDIT]: I will check this in the future, I am also interested in that. ATM quite busy with other projects, so this is in the queue.

  5. Interesting post. I’m a bit lost with the vmx distribution, but I think Im not alone.
    I’ve got different realease from juniper, the last one was 14.1-20150323. My contact at juniper say that is the latest, but I doubt. I was marked GA, so perhaps.

    Vmware support is mandatory for me, since this is the standard in my compagny (and honestly this is the best hypervisor for now). I will try to make this working with the local pfe. But the real interest is to make the phase 2 working. Is someone have ever succeed running it on vmware ?

    Jack : where are you find vMX14.2-MPC2-vm.vmdk ? in my distribution I ve got vpfe and vpfe-lite image..

    All of us : are you on irc or whatever to discuss further about this ?

    Thks.

  6. Hi,
    have you tested MVPN (rsvp ,ldp p2mp) ?

    I have vMX in VirtualBox, signaling seems to be ok, sender PE shows traffic is being send but on physical interfaces counters are zero.

  7. Dears
    First would like to thank you for the great document.

    May you please advise how can you edit /boot/loader.conf as i am a new in theee stuff

    Many thanks
    [Edit]: Hi, you need to use vi editor.

  8. Hi
    Anyone managed to fire phase 2 … I am not seeing em or ge interface on both RE & MPC vm
    Is this normal ?

    Thanks

  9. Hi,

    Thanks a lot for the the procedure . I’m facing some difficulties on the ge-* interfaces appear online , but i can’t seems to forward traffic , strange is that the interface counters shows packets passing in and out but i don’t have any ARP entry and no packets are actually sent.

    I’m using VirtualBox 4.3 and GNS3 1.3.7. , can you help me with this issue .. Tried virtino-net and E1000 interfaces but with no luck

    Many thanks in advance

  10. Hello. Excellent article, thank you for the information. I am running the vMX in virtualbox and it is great. Only thing I’m seeing is that the routing engine has a lot of cpu utilization by the kernel when I run 2 or more simultaneous vMX. In the processes I can see that riot is the one consuming more. Would you happen to know why? Thank you!

  11. I’m using Phase 1 vMX on VirtualBox. I was trying to setup simple L2Circuit and it works only using LDP LSPs. I wasn’t able to get it to work using RSVP because I can’t see any RSVP hellos being exchanged and no neighborships are being established. Did anyone face the same issue? I wonder if that’s a virtualbox problem or not.

      • Sorry that was my mistake. I forgot to add traffic-engineering or no-cspf. I should’ve paid more attention to the config before posting a comment. Thanks!

  12. Hello all,
    I’m facing an issue with my vMX 14.1R9, the image loads correctly, but takes over 30mins for the fpc 0 pic 0 to be detected. I’m using Ubuntu 15 and GNS3 1.3.8, qemu 2.2 and the qemu binary Im using is qemu-system-i386. Any ideas how to get it to work correctly? BTW, I have already performed the boot/loader.conf configuration but am still having trouble to get this working correctly.

  13. I’m using esxi 5.5 and the vm_local_rpio=”1″ is not working and I don’t see the ge-0/0/0 interfaces at all! Help!!

    • What vmx version are you running ? It does not work on 14.2. THis tutorial works only for 14.1R4.
      What is the output of “show chassis fpc pic-status” ?

      Andrei

  14. Update: I have managed to use 14.2R3-S3 with Phase2 (VM for RE and separate VM for MPC0) on VirtualBox (esxi works also):
    root@VMX4-re0> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis VMXcac5 MX960
    Midplane
    Routing Engine 0 RE-VMX
    CB 0 VMX SCB
    CB 1 VMX SCB
    FPC 0 Virtual 20x1G + 4x10G FPC
    CPU Rev. 1.0 RIOT 123XYZ987
    MIC 0 Virtual 20x 1GE(LAN) SFP
    PIC 0 BUILTIN BUILTIN Virtual 10x 1GE(LAN) SFP
    PIC 1 BUILTIN BUILTIN Virtual 10x 1GE(LAN) SFP
    MIC 1 Virtual 4x 10GE(LAN) XFP
    PIC 2 BUILTIN BUILTIN Virtual 2x 10GE(LAN) XFP
    PIC 3 BUILTIN BUILTIN Virtual 2x 10GE(LAN) XFP

    root@VMX4-re0> show interfaces ge* terse
    Interface Admin Link Proto Local Remote
    ge-0/0/0 up up
    ge-0/0/1 up up
    ge-0/0/2 up up
    ge-0/0/3 up up
    ge-0/0/4 up up
    ge-0/0/5 up up
    ge-0/0/6 up up
    ge-0/0/7 up up
    ge-0/0/8 up up
    ge-0/0/9 up up
    ge-0/1/0 up up
    ge-0/1/1 up up
    ge-0/1/2 up up
    ge-0/1/3 up up
    ge-0/1/4 up up
    ge-0/1/5 up up
    ge-0/1/6 up up
    ge-0/1/7 up up
    ge-0/1/8 up up
    ge-0/1/9 up up

    root@VMX4-re0> show version
    Hostname: VMX4-re0
    Model: mx960
    Junos: 14.2R3-S3
    JUNOS Base OS boot [14.2R3-S3]
    JUNOS Base OS Software Suite [14.2R3-S3]
    JUNOS Online Documentation [14.2R3-S3]
    JUNOS Crypto Software Suite [14.2R3-S3]
    JUNOS platform Software Suite [14.2R3-S3]

    I will add a post in the following weeks if anyone interested.

    • Hi Andrei,

      Can you please post a Phase 2 tutorial for ESXi too, but for vMX 14.1R5?

      Do you know if Juniper will release the the nearest future an official vMX version for VMWare?

      Thank you

    • Yep I’m also interrested to test vmx with phase2 on vmware. I’ve tried a little but have no luck with pfeimg on linux. They are booting, but the screen remains blank.

      • I would be interested in Phase 2 as well because it’s not working for me for some reason. After booting MPC2 VM I’m getting an error about creating em interfaces. I’m using 10x E1000 adapters on ESXi. It goes into Online testing mode just to turn Offline after a minute or so and it stays like that. My vSwitch is configured with 9000 MTU and I have promiscuous mode enabled.

        • VMX Phase 2 starts of with 14.2 which I’m having some issues with (vmxt and rpio daemons on MPC VM crash and the RE-VM detects the FPC as offline). Not clear to my why this is happening.

  15. Has anyone managed to power up Phase 3 vMX on Virtualbox or ESXi? RE boots up fine but I can’t get vPFE to work. It is constantly throwing the same error that it cant initialize NIC 0 card.

  16. Version 14.1R5.4
    ESXi 5.1

    i have done the vm_local_rpio=”1″
    the vm powers up and can login
    if i configure IP address on fxp0 i get no response
    if i configure IP address on em2 i get no response but if i ping gateway from vMX i get a response with a tonne of DUP messages when there is no DUP addresses.

    not seeing any ge-0 interfaces at all

    any advice?

    • Intel driver in FreeBSD that Junos on vMX is based on is not working properly on ESXi so even if you will setup your vMX you will have problem with the interfaces. I tried to deploy vMX on ESXi with success but it was dropping the interfaces anyway. Do yourself a favor and use KVM. I’m running 10 vMX instances on KVM on my laptop with Intel i5-4300U and 16Gb of RAM with 20-40% CPU utilization and 12.5 Gb of RAM leaving plenty of resources for few more vMX’s or hosts for the lab.

  17. Many thanks for this great article!

    It worked out of the box with 14.1R5.4 (no luck with ver. 15 so far).
    There is a problem, like it was mentioned already – high CPU load caused by riot process:

    root@% top -i
    last pid: 3650; load averages: 16.20, 38.50, 34.62 up 0+00:23:47 23:06:17
    395 processes: 2 running, 393 sleeping
    CPU states: 28.4% user, 0.0% nice, 70.8% system, 0.8% interrupt, 0.0% idle
    Mem: 531M Active, 107M Inact, 159M Wired, 431M Cache, 214M Buf, 6654M Free
    Swap: 6756M Total, 6756M Free

    PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
    3625 root 323 131 0 574M 14024K RUN 0:10 33.28% riot

    root> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis VM56367F8690 VMX
    Midplane
    Routing Engine RE-VMX
    CB 0 VMX SCB
    FPC 0 Virtual FPC
    CPU Rev. 1.0 RIOT 123XYZ987
    MIC 0

    root> show chassis routing-engine
    Routing Engine status:
    DRAM 8172 MB (8192 MB installed)
    Memory utilization 14 percent
    CPU utilization:
    User 90 percent
    Background 0 percent
    Kernel 10 percent
    Interrupt 0 percent
    Idle 0 percent
    Model RE-VMX
    Serial ID VM56367F8690
    Start time 2015-11-01 22:43:00 UTC
    Uptime 24 minutes, 22 seconds
    Last reboot reason panic:page fault
    Load averages: 1 minute 5 minute 15 minute
    14.70 32.42 32.60

    root@% cat /boot/loader.conf | grep rpio
    vm_local_rpio=”1″

    It’s running on licensed ESXi 5.5.
    Can someone advise where to dig?

    @mandrei05, can you please post how you managed to build separate RE?
    [Andrei] – In next few months, I’ll try to make a post for RE/PFE phase2 vMX. Will post everything there.

Leave a Reply

Your email address will not be published. Required fields are marked *


× six = 6

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>