The VMX is Juniper’s latest product in it’s virtual products portfolio. Official page is https://www.juniper.net/us/en/products-services/routing/mx-series/vmx/.
Update: Juniper Virtual MX (vMX) 14.2R5.3 Phase 2 (RE and MPC on different VMs).
At the time of this article, VMX is not released yet, but it can be obtained for beta testing through sales channel so you can request to IMG image which is qcow2 QEMU file format for usage with KVM hypervisor.
The virtual MX deployment had two phases:
Phase 1 based on Junos 14.1 having the Routing Engine and the virtual PFE in the same virtual machine (with a note that releases above 14.1R4 need a tweak to enable the local PFE instead of using a separate VM)
Phase 2 based on Junos 14.2 splitting the RE and PFE in different virtual machines and using an IP connection between them for communication.
I’m going to provide a short tutorial for deploying Phase 1 vMX running 14.1R4.9 with the tweak to enable the local virtual PFE (in the same VM).
To install the virtual MX router under VMWare ESXi hypervisor, we need to convert the qcow2 img vmx image, that it is shipped at this time, to vmdk disk format (vmware specific).
Converting vMX qcow2 / img disk to VMWare vmdk format
Copy the IMG vmx disk file on a QEMU installed system and run the following command to convert to vmdk:
# qemu-img convert -f qcow2 -O vmdk jinstall-vmx-14.1R4.9-domestic.img VMX-P6-14.1R4.9.vmdk
Setting the VMWare ESXi host to import the newly created VMDK disk
Once the VMDK disk is created, it needs to be copied to the ESXi host. Since this will be virtual machine disk that will get imported, we need to set up a special directory to reflect the VM name. In my case, I use it as P6 router, so I will make a /vmfs/volumes/datastore1/VMX-P6-14.1R4 directory for this vMX and copy the VMX-P6-14.1R4.9.vmdk disk inside it.
Creating the vMX virtual machine in vsphere client
In this step, it is required to create a new virtual machine and follow the next steps:
– Use a “Custom” configuration.
– Provide the name of the VM: “VMX-P6-14.1R4″ in my case.
– Select resource pool or use the default.
– Select the destination datastore for VM files.
– Select “Virtual Machine Version: 8″
– Select “Other” and “Other (64-bit)” operating system.
– Select at least 2 CPUs for the VM (one virtual socket and 2 cores per virtual socket in my case).
– Select 1 GB RAM memory.
– Add 4 NICs with E1000 adapter.
– Select “LSA Logic SAS” adapter.
– Chose “Use an existing virtual disk”.
– Browse the ESXi localstore holding the above VMDK disk file.
Note: At this stage it is important to know that vMX has 4 interfaces: em0, em1, em2 and em3. VMX will map em2 to ge-0/0/0, em3 to ge-0/0/1 and so on until ge-0/0/9 so the 3rd interface assigned to the virtual machine (em2) should be connected to the ESXi vSwitch that connects to the management LAN to provide access.
First steps to set up the vMX running 14.1R4.9 Junos
Using the vSphere client, power on and open the vMX console and wait for it to provide the login prompt. Login as “root” and you will be logged to the BSD shell.
Use “cli” command to go into Junos CLI and “conf” to start the configuration.
Issuing the “run show chassis fpc pic-status” from Junos configuration mode, you will see that the Virtual PFE is in “Present” state, not “Online” as it should. After a few minutes it will go into “Offline” state.
This is because vMX based on 14.1R4+ will try to connect to a remote PFE (different virtual machine) using IP subnet 172.16.0.1/16 on em1 interface used for RE and distributed PFE communication, but I do not have a distributed PFE so I want to use local PFE. To enable it, add vm_local_rpio=”1″ line to /boot/loader.conf and save the file. This will take effect after next reboot.
While still in configuration mode, we need to enable ge-0/0/0 interface to hook into traffic coming on the 3rd interface (em2). To do so, ge-0/0/0 interface needs static mac assignment as the mac of em2 interface. Also ge-0/0/0 will need an IP address from the management LAN.
The next step is to set a root password because without it, Junos will not allow any commits.
Now reboot vMX with “run request system reboot” and confirmation.
After reboot, the virtual PFE will be “Online” and it will bring up ge-0/0/0 to ge-0/0/9 interfaces and the vMX will be accessible via ssh:
$ ssh 10.1.1.156
The authenticity of host '10.1.1.156 (10.1.1.156)' can't be established.
ECDSA key fingerprint is 6d:b1:01:0b:af:0f:ea:26:ba:6b:a0:66:ad:80:ce:8a.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.156' (ECDSA) to the list of known hosts.
Password:
Password:
--- JUNOS 14.1R4.9 built 2015-02-01 01:53:57 UTC
root@%
root@% cli
root> show chassis fpc pic-status
Slot 0 Online Virtual FPC
PIC 0 Online Virtual 10x1GE PIC
root> show interfaces ge* terse
Interface Admin Link Proto Local Remote
ge-0/0/0 up up
ge-0/0/0.0 up up inet 10.1.1.156/24
multiservice
ge-0/0/1 up up
ge-0/0/2 up up
ge-0/0/3 up up
ge-0/0/4 up up
ge-0/0/5 up up
ge-0/0/6 up up
ge-0/0/7 up up
ge-0/0/8 up up
ge-0/0/9 up up
root> show interfaces em* terse
Interface Admin Link Proto Local Remote
em1 up up
em1.0 up up inet 172.16.0.1/16
inet6 fe80::20c:29ff:fe2a:c1b7/64
em2 up up
em3 up up
root> show interfaces fxp* terse
Interface Admin Link Proto Local Remote
fxp0 up up
Let’s discuss the above output:
– Virtual FPC0 (the local virtual PFE is online).
– 1 there are 4 interfaces assigned to the vMX virtual machine and they are mapped in the following manner:
1st NIC – fxp0 – management interface hooked directly into the Routing Engine
2nd NIC – em0
3rd NIC – em1 – Used for RE to distributed PFE communication
4th NIC – em2 – ge-0/0/0
…
13th NIC – em11 – ge-0/0/9
I used ge-0/0/0 here for the management lan and not fxp0 interface to make the interface mapping more visible, but fxp0 interface should be used for management, so let’s fix that:
[edit]
root> configure
Entering configuration mode
[edit]
root# delete interfaces ge-0/0/0
[edit]
root# set interfaces fxp0.0 family inet address 10.1.1.156/24
[edit]
root# set system host-name P6
[edit]
root# commit
commit complete
[edit]
root@P6#
At this point, virtual MX is set up, has a hostname P6, uses 1st interface as fxp0 as an RE interface, 2nd interface as em0 (not clear to me atm what it is used for), 3rd interface as em1 used for RE / PFE communication, 4th -to 13th interfaces as em2-em11 hooked into the PFE mapped as ge-0/0/0 to ge-0/0/9 interfaces once these ge* interfaces are configured to used the mac addresses of the corresponding em2+ interfaces.
vMX is intended to have feature parity with the Juniper’s physical MX 3D router and I have tested to work:
– BGP RR, iBGP, eBGP, multipath, Add-path.
– LDP, RSVP, IGP TE, RSVP-TE, CSPF, VPLS, L3VPN, L2VPN, EVPN.
– virtual bridges.
Flowspec I havent tested.
[Admin Edit]: I have created following forum for vMX related discussions: https://forum.ivorde.com/juniper-virtual-mx-vmx-f146.html
Hi
DO you know how connect MPC2 MODULE TO ( vMX14.2-MPC2-vm.vmdk) TO RE0 VIRTUAL MACHINE ?
the new VMX 14.2 has been released as well
I think they connect via em1 interface. They should have layer 2 reachability, but I’d have to test that in the future to be sure.
Hi, did you have any 100% CPU usage issue? I can’t get rid of it. It works, but vMX constantly consumes whole CPU power.. Any ideas how to avoid it?
Check your RE CPU utilization (show chassis routing-engine) and check your interfaces counters (> monitor interface traffic). Using OSPF/ISIS/BGP ?
There is “riot” process that consumes all the CPU power.. I have almost nothing in my configuration, no protocols, no services, no addresses etc. The same is when I use only one machine and enable not mentioned officialy vm_local_rpio=”1″ parameter in loader.conf.
RIOT process seems to be the software simulation of the LU chip in hardware MX. Can you post output from “top” command ?
is there any difference between the phase 1 and phase II in terms of features supported,
do u have any idea ??
Unfortunately I dont have a comprehensive list of features roadmap for VMX. Only thing I can see so far is the EVPN supporting the default gateway extended community starting with 15.2 and also per vlan label allocation starting from 14.1R3 instead of per EVI.
Hi,
is it possible to have two REs in vMX to test some of the redundancy options in juniper ? if yes pls let me know how to configure to have two REs with one MPC
[EDIT]: I will check this in the future, I am also interested in that. ATM quite busy with other projects, so this is in the queue.
Interesting post. I’m a bit lost with the vmx distribution, but I think Im not alone.
I’ve got different realease from juniper, the last one was 14.1-20150323. My contact at juniper say that is the latest, but I doubt. I was marked GA, so perhaps.
Vmware support is mandatory for me, since this is the standard in my compagny (and honestly this is the best hypervisor for now). I will try to make this working with the local pfe. But the real interest is to make the phase 2 working. Is someone have ever succeed running it on vmware ?
Jack : where are you find vMX14.2-MPC2-vm.vmdk ? in my distribution I ve got vpfe and vpfe-lite image..
All of us : are you on irc or whatever to discuss further about this ?
Thks.
Hi,
have you tested MVPN (rsvp ,ldp p2mp) ?
I have vMX in VirtualBox, signaling seems to be ok, sender PE shows traffic is being send but on physical interfaces counters are zero.
*sent
Alex, havent tested these features yet.
Hello, Have you tested Class of Service feature on vMX ? I tried it on 14.1R1 with no luck.
Dears
First would like to thank you for the great document.
May you please advise how can you edit /boot/loader.conf as i am a new in theee stuff
Many thanks
[Edit]: Hi, you need to use vi editor.
Many thx …it workrd like charm
Hi
Anyone managed to fire phase 2 … I am not seeing em or ge interface on both RE & MPC vm
Is this normal ?
Thanks
Hi,
Thanks a lot for the the procedure . I’m facing some difficulties on the ge-* interfaces appear online , but i can’t seems to forward traffic , strange is that the interface counters shows packets passing in and out but i don’t have any ARP entry and no packets are actually sent.
I’m using VirtualBox 4.3 and GNS3 1.3.7. , can you help me with this issue .. Tried virtino-net and E1000 interfaces but with no luck
Many thanks in advance
Answered by mail
Hello. Excellent article, thank you for the information. I am running the vMX in virtualbox and it is great. Only thing I’m seeing is that the routing engine has a lot of cpu utilization by the kernel when I run 2 or more simultaneous vMX. In the processes I can see that riot is the one consuming more. Would you happen to know why? Thank you!
I’m using Phase 1 vMX on VirtualBox. I was trying to setup simple L2Circuit and it works only using LDP LSPs. I wasn’t able to get it to work using RSVP because I can’t see any RSVP hellos being exchanged and no neighborships are being established. Did anyone face the same issue? I wonder if that’s a virtualbox problem or not.
I have 14.1R4.10 with RSVP LSPs working on VirtualBox without any problems.
Just checking: can you post output from show rsvp interfaces ?
Sorry that was my mistake. I forgot to add traffic-engineering or no-cspf. I should’ve paid more attention to the config before posting a comment. Thanks!
Hello all,
I’m facing an issue with my vMX 14.1R9, the image loads correctly, but takes over 30mins for the fpc 0 pic 0 to be detected. I’m using Ubuntu 15 and GNS3 1.3.8, qemu 2.2 and the qemu binary Im using is qemu-system-i386. Any ideas how to get it to work correctly? BTW, I have already performed the boot/loader.conf configuration but am still having trouble to get this working correctly.
I’ve seen VMX not to work correctly with GNS. Try Virtual Box.
qemu-system-i386 is a 32-bit emulator, you almost certainly need qemu-system-x86_64
I’m using esxi 5.5 and the vm_local_rpio=”1″ is not working and I don’t see the ge-0/0/0 interfaces at all! Help!!
What vmx version are you running ? It does not work on 14.2. THis tutorial works only for 14.1R4.
What is the output of “show chassis fpc pic-status” ?
Andrei
Update: I have managed to use 14.2R3-S3 with Phase2 (VM for RE and separate VM for MPC0) on VirtualBox (esxi works also):
root@VMX4-re0> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis VMXcac5 MX960
Midplane
Routing Engine 0 RE-VMX
CB 0 VMX SCB
CB 1 VMX SCB
FPC 0 Virtual 20x1G + 4x10G FPC
CPU Rev. 1.0 RIOT 123XYZ987
MIC 0 Virtual 20x 1GE(LAN) SFP
PIC 0 BUILTIN BUILTIN Virtual 10x 1GE(LAN) SFP
PIC 1 BUILTIN BUILTIN Virtual 10x 1GE(LAN) SFP
MIC 1 Virtual 4x 10GE(LAN) XFP
PIC 2 BUILTIN BUILTIN Virtual 2x 10GE(LAN) XFP
PIC 3 BUILTIN BUILTIN Virtual 2x 10GE(LAN) XFP
root@VMX4-re0> show interfaces ge* terse
Interface Admin Link Proto Local Remote
ge-0/0/0 up up
ge-0/0/1 up up
ge-0/0/2 up up
ge-0/0/3 up up
ge-0/0/4 up up
ge-0/0/5 up up
ge-0/0/6 up up
ge-0/0/7 up up
ge-0/0/8 up up
ge-0/0/9 up up
ge-0/1/0 up up
ge-0/1/1 up up
ge-0/1/2 up up
ge-0/1/3 up up
ge-0/1/4 up up
ge-0/1/5 up up
ge-0/1/6 up up
ge-0/1/7 up up
ge-0/1/8 up up
ge-0/1/9 up up
root@VMX4-re0> show version
Hostname: VMX4-re0
Model: mx960
Junos: 14.2R3-S3
JUNOS Base OS boot [14.2R3-S3]
JUNOS Base OS Software Suite [14.2R3-S3]
JUNOS Online Documentation [14.2R3-S3]
JUNOS Crypto Software Suite [14.2R3-S3]
JUNOS platform Software Suite [14.2R3-S3]
I will add a post in the following weeks if anyone interested.
Yes, would be very helpful
Hi Andrei,
Can you please post a Phase 2 tutorial for ESXi too, but for vMX 14.1R5?
Do you know if Juniper will release the the nearest future an official vMX version for VMWare?
Thank you
Yep I’m also interrested to test vmx with phase2 on vmware. I’ve tried a little but have no luck with pfeimg on linux. They are booting, but the screen remains blank.
I would be interested in Phase 2 as well because it’s not working for me for some reason. After booting MPC2 VM I’m getting an error about creating em interfaces. I’m using 10x E1000 adapters on ESXi. It goes into Online testing mode just to turn Offline after a minute or so and it stays like that. My vSwitch is configured with 9000 MTU and I have promiscuous mode enabled.
VMX Phase 2 starts of with 14.2 which I’m having some issues with (vmxt and rpio daemons on MPC VM crash and the RE-VM detects the FPC as offline). Not clear to my why this is happening.
I have some trouble find this vMX image on internet, can anybody give me a hint! Thanks a lot
[Admin]: You can download it from http://www.juniper.net/support/downloads/?p=vmx . This site does not provide Juniper images.
Has anyone managed to power up Phase 3 vMX on Virtualbox or ESXi? RE boots up fine but I can’t get vPFE to work. It is constantly throwing the same error that it cant initialize NIC 0 card.
does l2vpn, l3vpn and evpn works without license?
anyone tested
I’ve tested L2VPN and L3VPN so far and everything works as expected. It seems like there is no license required because I don’t have any and also show system license on Junosphere gives the same output.
Indeed, L2/3VPN works without license on 14.1 I see.
Hi, my vmx just see em interfaces, how can i change that?
VMX version ? Hypervisor model and version ? Did you modify loader.conf file and did you reboot ?
Hi,
I don’t see anywhere em0 interface.
For me it is
fxp0
em1
.
.
.
em11
Please advise, ver Junos: 14.1R5.4
fxp0 should be the management. There shouldn’t be any em0.
Hi,
Has anyone had any luck with the vPFE? Mine just seems to get stuck on booting.
Version 14.1R5.4
ESXi 5.1
i have done the vm_local_rpio=”1″
the vm powers up and can login
if i configure IP address on fxp0 i get no response
if i configure IP address on em2 i get no response but if i ping gateway from vMX i get a response with a tonne of DUP messages when there is no DUP addresses.
not seeing any ge-0 interfaces at all
any advice?
Intel driver in FreeBSD that Junos on vMX is based on is not working properly on ESXi so even if you will setup your vMX you will have problem with the interfaces. I tried to deploy vMX on ESXi with success but it was dropping the interfaces anyway. Do yourself a favor and use KVM. I’m running 10 vMX instances on KVM on my laptop with Intel i5-4300U and 16Gb of RAM with 20-40% CPU utilization and 12.5 Gb of RAM leaving plenty of resources for few more vMX’s or hosts for the lab.
Many thanks for this great article!
It worked out of the box with 14.1R5.4 (no luck with ver. 15 so far).
There is a problem, like it was mentioned already – high CPU load caused by riot process:
root@% top -i
last pid: 3650; load averages: 16.20, 38.50, 34.62 up 0+00:23:47 23:06:17
395 processes: 2 running, 393 sleeping
CPU states: 28.4% user, 0.0% nice, 70.8% system, 0.8% interrupt, 0.0% idle
Mem: 531M Active, 107M Inact, 159M Wired, 431M Cache, 214M Buf, 6654M Free
Swap: 6756M Total, 6756M Free
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
3625 root 323 131 0 574M 14024K RUN 0:10 33.28% riot
root> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis VM56367F8690 VMX
Midplane
Routing Engine RE-VMX
CB 0 VMX SCB
FPC 0 Virtual FPC
CPU Rev. 1.0 RIOT 123XYZ987
MIC 0
root> show chassis routing-engine
Routing Engine status:
DRAM 8172 MB (8192 MB installed)
Memory utilization 14 percent
CPU utilization:
User 90 percent
Background 0 percent
Kernel 10 percent
Interrupt 0 percent
Idle 0 percent
Model RE-VMX
Serial ID VM56367F8690
Start time 2015-11-01 22:43:00 UTC
Uptime 24 minutes, 22 seconds
Last reboot reason panic:page fault
Load averages: 1 minute 5 minute 15 minute
14.70 32.42 32.60
root@% cat /boot/loader.conf | grep rpio
vm_local_rpio=”1″
It’s running on licensed ESXi 5.5.
Can someone advise where to dig?
@mandrei05, can you please post how you managed to build separate RE?
[Andrei] – In next few months, I’ll try to make a post for RE/PFE phase2 vMX. Will post everything there.
I would like to know how did you manage to add a second fpc?? and to add many pics in a fpc???
training
check the routing protocols you are running specially isis ./.deactivate them and see performance
Activating some protocols does seem to have a small impact on CPU.