DevConf.cz 2016

The workshop init script 'vm_env_setup.sh' is in http://github.com/arcolife/latency_analyzer/

So, this is an ongoing investigation of KVM blockIO event tracing and analysis, within the performance engineering team at Red Hat. During this process, we have come come across a few anomalies which we'd like to share with the community to gain support and contribution for tooling/kernel modules of Linux, associated with performance. We have, as a part of this investigation, also released a couple of tools, which we'd like to showcase at DevConf.

This talk is intended for system admins as well as those seeking general performance tuning/analysis. The lab would be a mix of a brief overview followed by a hands on tracing of events, analysis of a test case and reaching conclusions based on that result.

The project link is a work in progress but we have released some utilities and will continue to work on the following repositories as well:
- http://github.com/psuriset/kvm_io/
- http://github.com/arcolife/perf-script-postprocessor

Please note that vm_env_setup.sh runs perfectly on fedora 23. If you have other distros/versions, kindly at least do the following, to speed up the workshop:

1. install the pip2 module perf-script-postprocessor. You might get dependency erros on rpm based systems. So install the equivalent of following packages.

   gcc lapack lapack-devel blas blas-devel gcc-gfortran gcc-c++ liblas libffi-devel libxml-devel libxml2-devel libxslt-devel redhat-rpm-config

2. install @Virtualization packages for your distro, as well as qemu-kvm ..so we could use virsh / virt-install / qemu-kvm as accelerator..

3. run the following part from vm_env_setup.sh, as following..

   # ./handy_minimalistic.sh

Cheers.

 

The oVirt project allows efficient management of virtualized
datacenters. Deciding what machine should host a certain VM is one of
the important functions of the management platform. Unfortunately it
is also one of the complex ones, because there can be many rules
governing the placement policy and there is a time limit in which a VM
has to be started or migration initiated.

All major projects today have a similar scheduling scheme where each VM is
considered separately and that causes fragmentation of free resources.
Both Nova scheduler and Kubernetes provide a set of filter and weight modules.

oVirt has a rich set of modules too but we are now allowing
integration with a next
generation algorithm set based on probabilistic (soft computing) methods thanks
to the Optaplanner project.

In order to support highly available VMs, oVirt needs a mechanism to determine the status of a VM running on non-responsive host. We need to be sure that VM is not running before restarting it on a different host to prevent data corruption. And oVirt fencing flow is crucial to achieve this goal.

Host fencing flow is a part of oVirt host monitoring, it takes care of non-responsive hosts and tries to make them responsive again. We will describe each step of the whole fencing flow on real world issues, describe what actions around hosts/VMs are taken after each step of fencing flow and also describe how power management interface of the hosts is used.

This talk is about virtualization with KVM and QEMU on the powerpc platform, especially with regard to the new POWER8 systems. After a short introduction to the Power platform in general, the differences to KVM on the x86 platform will be explained, like the devices that are special to the PPC sPAPR platform and the different kind of KVM kernel modules. To give some help with troubleshooting a non-working guest partition, we'll also have a look the user-related concepts of the guest boot firmware ("Open Firmware" as it is called in the powerpc world).

Microsoft Hyper-V is among supported hypervisors for Linux kernel since 2009. What's actually supported and why do we need all the specifics in kernel? What are the PV drivers and how do they work? In my presentation I'll try to cover the internals of Hyper-V support in Linux kernel, current status of different drivers and challenges we face trying to make Linux work even better.

Virtual machines and containers are everywhere. They have become an important part of not only developer's live but mainly system administrators and whole enterprise environment. With that huge impact, the performance is becoming more important. In my talk I am going to show various approaches since performance is tied to given use case. While some users have some dedicated machines for virtualization and can isolate most of their CPUs for virtual machine's sake, usual developers on desktops almost certainly will not do that. And while I/O throughput is important for databases running in virtual machines, rendering a video aims at something else: vCPU and memory throughput. I'm going to cover these in my talk. Moreover, the use cases will be covered with live demos too.

This talk will present how Windows builds of SPICE components are made, with the goal of showing that this is quite easy, and can be done from the comfortable familiarity of a Linux machine. A short overview of current and upcoming SPICE features will also be given.

www.spice-space.org is a link for the SPICE project.

A presentation about a design for a next generation config management tool, and the specific problems this design solves.

Three of the main design features of the tool include:
* Parallel execution
* Event driven mechanism
* Distributed architecture

This talk will demo a prototype I've built that implements these ideas. It is written in golang, and is completely free software.

Pre-copy live migration refers to the process of moving a running virtual machine or application between dierent physical machines without disconnecting the client or application. A live migration keeps the guest running on the source host and begins moving the memory without stopping the guest. All modied memory pages are monitored for changes and sent to the destination while the image is sent. The memory is updated with the changed pages. The process continues until the amount of pause time allowed for the guest equals the predicted time for the nal few pages to be transferred.

A problem with pre-copy live migration is that depending on the access pattern in the guest system, the migration process might be unable to reduce the number of dirty pages below the number necessary for the congured pause time to be met.

This presentation analyzes the optimal pattern to copy memory assuming the guest access pattern is cyclic and known in advance.

Qemu can use two different methods to submit disk I/O requests. The default method, "io=threads", uses a thread pool to submit these requests synchronously, while the alternate method, "io=native", uses a single thread to submit these requests asynchronously. We were curious if the default method provided the best overall throughput, so we tested both under a wide array of storage configurations. This included the use of solid state and rotational disks, different host filesystems, and even using a network file system. This paper consists of our analysis results, challenges and improvements.

Continuous integration (CI) is a practice where isolated changes
and/or patches are immediately tested and reported on when they
are added either upstream or downstream. The goal of CI is to
provide rapid feedback so that if a defect is introduced into the
code base, it can be identified and corrected as soon as possible.
The libvirt and qemu communities are promoting the use of CIand
are working on integrating Jenkins with test automation frameworks
such asAvocado and OpenStack to make the process more efficient.
This presentation will cover recentdevelopments in the CI efforts
by these communities. We will also discusssome of the issues faced
(and fixed) during deployment of theinfrastructure, pitfalls of
cross-platform integration of effortsand a planned road-map and
next steps to sustain a developingenvironment for all future releases.

Virtualization drivers (e.g. libvirt, QEMU/KVM) are the core part of OpenStack Compute layer. An OpenStack environment is challenging to debug as is -- more so when multiple Compute nodes and thereby multiple libvirt daemons and QEMU
instances are involved. A good grasp of Virtualization debugging mechanisms is vital for effective root cause analysis. To that end, libvirt and QEMU provide a rich set of debugging controls that allow us to query (or modify) the state of virtual machines in distress.

This talk focuses on providing an in-depth view of aforementioned techniques.

Topics include: debugging Nova Compute process crashes; gathering specific patterns from libvirt log filters, libvirt environment variables, and systemd journal fields; live querying the VM (and QEMU) state through `virsh` and QEMU Machine Protocol (QMP) commands; tuning the libvirt daemon logging; monitoring events emitted by QEMU, etc. Along with a real-world example that ties together some of the techniques discussed.

Audience would include OpenStack infrastructure operators, Virtualization (libvirt/QEMU/KVM) administrators, developers, tinkerers, or any one interested in understanding the Virtualization layer in OpenStack to help equip yourself with better debugging techniques.

Ovirt and Gluster Hyperconvergence architecture is a setup where oVirt and Gluster both will run in single commodity boxes that can scale horizontally. This talk will focus on the details of design, configuration and installation of hyper converged hosts with oVirt as the software stack and Gluster for the storage.

Brief Abstract:

Gluster is a popular distributed scale-out filesystem, and KVM is the de-facto hypervisor on linux.
We will talk about setting-up hyper-converged environment using Gluster and KVM.
This will also cover improvements done in Gluster for virtualization use case in general and hyper-converged use case in particular.

Some of the improvements include:

- Enabling Qemu's Gluster block backend for multiple storage servers
- Enhance the network disk support for gluster in libvirt
- Bug fixes for the virtualization use case

- Introduce transport over Unix domain Socket for hyper-converged setup
- Bug fixes In RDMA transport
- Improvements for dynamic authentication

Talk will end with demo showcasing hyperconvergence with gluster.

This talk will cover the integration of Ceph, an highly available
software defined storage, with oVirt virtualization platform.  The
integration is facilitated using Cinder, a block storage service for
OpenStack.

We will explore the differences between oVirt traditional storage and
the new Ceph storage type. In addition, learn how Ceph authentication
keys are managed,  introduce the new APIs added for managing live
snapshots for Ceph disks, and discuss future development.