== System Requirements: ==
Enable the following in the BIOS:
BIOS -> Advanced -> Advanced Chipset Settings
Intel VT-d [Enabled]
SR-IOV Supported [Enabled]
Intel I/OAT [Enabled]
BIOS -> Advanced -> Configure advanced CPU settings
Intel(R) Virtualization Tech [Enabled]
Note : If the “SR-IOV Supported” option is not there in the BIOS, the BIOS needs to be updated.
On SuperMicro X8 Motherboards, the BIOS Version needs to be >= 1.1a.
Supermicro X7 Motherboards won’t have the SR-IOV Option even if the BIOS is updated.
== Setting up KVM: ==
1.Install RHEL 6.x 64-bit OS.
2.Compile 3.6.11 kernel on the above machine. Use the following files for compilation:
(a)Linux-3.6.11.tar.bz2 [wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.6.11.tar.bz2].
(b).config file for the compilation[/home/sonyarpitad/vNIC_Conf/config_3.6.11].
3.After the 3.6.11 kernel is successfully compiled, make the following entry in the /boot/grub/menu.lst file and boot into 3.6.11 kernel:
title Red Hat Enterprise Linux Server (3.6.11)
root (hd0,0)
kernel /vmlinuz-3.6.11 ro root=LABEL=/1 intel_iommu=on
initrd /initrd-3.6.11.img
4.Boot into 3.6.11 Kernel and verify if Intel IOMMU is enabled using “dmesg | grep –i iommu”. If IOMMU is not enabled, the Link won’t come up on the VMs.
5.Install “qemu-kvm-1.2.0.tar.gz” [/home/sonyarpitad/vNIC_Conf/qemu-kvm-1.2.0.tar.gz] package to enable qemu.
== Setting up VM: ==
1.Boot into 3.6.11 kernel.
2.Load the kvm and kvm-intel or kvm-amd
Example: #modprobe kvm allow_unsafe_assigned_interrupts=1
#modprobe kvm-intel
3. Since kvm module gets loaded automatically due to dependency of kvm_intel on it, add this to a conf file under modprobe.d as below:
# cat /etc/modprobe.d/kvm.conf
Options kvm allow_unsafe_assigned_interrupts=1
4.Create disk image for the VM using the command:
# qemu-img create -f qcow2 vdisk.img <size of disk image for VM>
[example-- qemu-img create -f qcow2 vdisk.img 30G]
<u>'''NOTE'''</u>: User needs to follow the VNC configuration in case the he/she is working on a remote computer and not infront of the console of the Test computer.
== VNC Configuration: ==
1.Copy VNC configuration file[/home/sonyarpitad/vNIC_Conf/xstartup] to /root/.vnc/
2.Start VNCServer :
# vncserver -geometry 1440x900 :1
3. After the VNC server is started, open a VNC session to work on the VMs. [Can use any VNCviewer]
== Installing OS on the VM: ==
1.Insert the CD or use ISO for the respective OS to be installed on the VM on the physical host.
2.(a)Execute the following command in a VNC session to bring up the VM while booting from a cd-rom:
#qemu-system-x86_64 -hda vdisk.img -cdrom /dev/cdrom -boot d –smp 2 -m 1024 &
If the ISO image is available, copy it to the Physical Host and install the OS in VM using:
#qemu-system-x86_64 -hda vdisk.img -cdrom <path to boot-media.iso> -boot d –smp 2 -m 1024 &
where,
-hda use 'file' as IDE hard disk image
-cdrom use 'file' as IDE cdrom image
- boot [order=drives][,once=drives][,menu=on|off]
'drives': floppy (a), hard disk (c), CD-ROM (d), network (n)
-smp n set the number of CPUs to 'n' [default=1]
-m megs set virtual RAM size to megs MB [default=128]
(b)Once the VM boots up, install OS on it. Once it asks for reboot, press “Reboot” and when the VM comes up close the “QEMU” window and boot the VM using the following command:
#qemu-system-x86_64 –hda vdisk.img –boot c –smp 2 –m 1024 –net none &
where,
-net none use it alone to have zero network devices
(c)Complete the installation procedure and reboot the VM. Poweroff the VM.
3.Setting corporate network on the VM:
(a)On the Physical host setup a bridge network.
#brctl addbr br0
#ifconfig eth0 0.0.0.0 up
#brctl addif br0 eth0
#ifconfig br0 <ip/netmask of the corporate interface>
Add route- #route add default gw <IP address of gateway>
(b)Start VM specifying a tap interface and a vitual nic.
#qemu-system-x86_64 –hda vdisk.img -boot –c –smp 2 –m 1024 -net nic,vlan=0,model=e1000 –net tap,ifname=tap0,vlan=0,script=no &
(c)While the VM is booting add the tap0 interface on the physical host to the bridge.
#ifconfig tap0 up
#brctl addif br0 tap0
== Bringing up the VM with T4 VF: ==
1.Load the cxgb4 driver on the Physical host specifying the VFs:
# modprobe cxgb4 num_vf=w,x,y,z
where w: number of VFs on PF0
x: number of VFs on PF1
y: number of VFs on PF2
z: number of VFs on PF3
2.Execute “lspci” on the host and note the PCI ID assigned to the VM [Example:04:01.1]
3.Power up the VM attaching the VF:
# qemu-system-x86_64 –hda vdisk.img -boot –c –smp 2 –m 1024 -net nic,vlan=0,model=e1000,mac=00:xx:yy:zz:ww:vv –net tap,ifname=tap1,vlan=0,script=no –pcidevice host=<PCI ID of the VF> &
where xx,yy,zz,ww,vv are hex values
While the VM is booting add the tap0 interface to the bridge.
#ifconfig tap1 up
#brctl addif br0 tap1
4.When the VM is brought up, load the cxgb4vf driver.
5.Execute “ifconfig -a” to make sure that T4 VF is detected on the VM.
6.Bring up the VF: ifconfig ethX up
== Serial Console on VM: ==
1. # qemu-system-x86_64 –hda vdisk.img -boot –c –smp 2 –m 1024 -net nic,vlan=0,model=e1000,mac=00:xx:yy:zz:ww:vv –net tap,ifname=tap1,vlan=0,script=no –pcidevice host=<PCI ID of the VF> -serial file:<dump-filename> &
2. On VM /boot/grub/menu.lst add the following entry:
console=ttyS0,115200
Refer: http://www.linux-kvm.org/page/Bugs
1.12.25 kernel.org
1.11.27.0 suse,rhel inbox
rmmod kvm_intel
rmmod kvm
modprobe kvm allow_unsafe_assigned_interrupts=1
modprobe kvm_intel
rmmod cxgb4
modprobe cxgb4 num_vf=4,0,0,0
. bridge.sh
qemu-system-x86_64 -smp 4 -m 4048 -hda /root/vnic/vdisks/vdisk_3_6_11.img -net nic,model=e1000,macaddr=00:11:22:33:44:55 -net tap,ifname=tap2,vlan=0,script=no,downscript=no -device pci-assign,host=03:01.0 &
ifconfig tap2 up
brctl addif br0 tap2
How to configure VLAN -
=====================
ip link add link ens4 name ens4.23 type vlan id 23
ip link set ens4.23 up
ip addr add 102.23.23.74/24 dev ens4.23
How to bring link up/down -
=========================
ip link set ens4 up
ip link set ens4 down
How to assign IP -
================
ip addr add 102.1.1.74/24 dev ens4
How to assign default gw -
========================
ip route add default via 10.192.192.1
How to change MTU -
=================
ip link set mtu 9000 dev ens4
VM IPs -
------
10.192.194.72/22
10.192.194.73/22
74
75
gw - 10.192.192.1
qemu-system-x86_64 -smp 4 -m 4048 -hda /root/vnic/vdisks/vdisk_3_6_11.img -net nic,model=e1000,macaddr=00:11:22:33:44:55 -net tap,ifname=tap2,vlan=0,script=no,downscript=no -device pci-assign,host=03:01.0 -device pci-assign,host=03:01.4 &
qemu-system-x86_64 -smp 4 -m 4048 -hda /root/vnic/vdisks/vdisk_3_6_11.img -net nic,model=e1000,macaddr=00:11:22:33:44:55 -net tap,ifname=tap2,vlan=0,script=no,downscript=no -device pci-assign,host=03:01.0 -device pci-assign,host=03:01.1 &
brctl addif br0 tap2
ifconfig tap2 up
logs:
=====
cat /sys/kernel/debug/cxgb4/0000\:03\:00.4/devlog > devlog
cat /sys/kernel/debug/cxgb4/0000\:03\:00.4/edc0 > edc0
cat /sys/kernel/debug/cxgb4/0000\:03\:00.4/edc1 > edc1
cat /sys/kernel/debug/cxgb4/0000\:03\:00.4/mc1 > mc1
qemu-system-x86_64 -smp 4 -m 4048 -hda /root/vnic/vdisks/vdisk_3_6_11.img -net nic,model=e1000,macaddr=00:11:22:33:44:55 -net tap,ifname=tap2,vlan=0,script=no,downscript=no -device pci-assign,host=03:01.0 &
ifconfig tap2 up
brctl addif br0 tap2
qemu-system-x86_64 -smp 4 -m 4048 -hda /root/vnic/vdisks/vdisk-RHEL7u1-Snapshot.img -net nic,model=e1000,macaddr=00:11:22:33:44:55 -net tap,ifname=tap3,vlan=0,script=no,downscript=no -device pci-assign,host=03:01.4 &
ifconfig tap3 up
brctl addif br0 tap3
fallocate -l2g junk
03:01:0
03:01:1
4
5
- - - -
01:0 0 1 2 3
4 5 6 7
02:0 - - - -
0 1 2 3
4 5 6 7
vf's of PF0| vf's of PF1
PPC8:
=====
3.17.8
No comments:
Post a Comment