Providing Solutions For Life

Vagrant on steroids


This is our life when we are working on automating some really complicated machine building and provisioning procedures. I.e. while developing the playbooks in Ansible, or CookBooks in Chef or manifests in Puppet.

It is not easy to fail faster and fix early as the script might have to download all the dependencies again and again. Adding to all the odds, it could be that the dependencies are being downloaded on a low bandwidth.

Even though Vagrant makes bringing up VM and their management faster, the provisioning (using Ansible, Chef or Puppet or etc) might take inordinately long times when it involves, and it usually does involve downloading packages on the VMs. And it gets painful when you have to download a full stack of several libraries to test your VM.

To help overcome this issue, we can use the following to cache the dependencies and test the configuration scripts or recipes faster, the following should set you up (assuming you have vagrant already installed):-

1. Install vagrant-cachier plugin


vagrant plugin install vagrant-cachier

2. Install vagrant-proxy plugin

A Vagrant plugin that configures the virtual machine to use specified proxies.

vagrant plugin install vagrant-proxyconf

3. Install a proxy server


I am on mac and hence using squidman to install it use the below one line command

sudo brew cask install squidman



Configuring to use them


1. Configuring to use the vagrant-cachier.


Edit the Vagrantfile to include these line

 if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope              = :box
    config.cache.synced_folder_opts = {
        type:          :nfs,
        mount_options: ['rw', 'vers=3', 'tcp', 'nolock']
    }
  end 

2. Configuring vagrant-proxyconf

Change the IP appropriately

if Vagrant.has_plugin?("vagrant-proxyconf")
# Start squid-man Mac OSX proxy on port 8081
# config.proxy.enabled = false
config.proxy.http     = "http://10.211.55.2:8081"
config.proxy.https    = "http://10.211.55.2:8081"
config.proxy.no_proxy = "localhost,127.0.0.1"
end 

3. Launch SquidMan and configure

As in the screenshots below. General tab, change the IP appropriately.

Clients tab add the following depending on your subnet
And start the Squidman


Confirming that it is all working

1. If you are on Ubuntu for example you would start seeing a lot of packages in the following folder
~/.vagrant.d/cache/parallels/ubuntu-14.04/apt/

2. Run the vagrant up and check squid's access logs. (Open Squidman and press Command+T and click on "Access log")

3. After these configurations have been put in place, Provisioning gets faster from the second time onwards. As for the first time, the dependencies get cached.

And for dependencies like JDK (which don't get cached due to https) you might want to create an image by snapshotting the VM with the OS and these dependencies. This will allow you to develop recipes you can skip their installations every time you run the palybooks or cookbooks


Hope this helps. (I am finally going to hit the Publish button :) after so many days of slacking on this. However do let me know if you would need clarity on anything)


Ansible - Error - stderr: E: There are problems and -y was used without --force-yes

In case your tasks is to install some packages and it errors out as below

  

- name: Install linux-headers
  apt: pkg={{item}} 
       state=installed 
       install_recommends=yes 
       update_cache=yes
  with_items: 
      - linux-headers-generic
      - dkms
  sudo: yes




failed: [parallelsUbuntu] => (item=linux-headers-generic,dkms) => {"failed": true, "item": "linux-headers-generic,dkms"}
stderr: E: There are problems and -y was used without --force-yes

stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  cpp fakeroot gcc libfakeroot linux-headers-3.13.0-63
  linux-headers-3.13.0-63-generic patch
Suggested packages:
  cpp-doc dpkg-dev debhelper gcc-multilib manpages-dev autoconf automake1.9
  libtool flex bison gdb gcc-doc diffutils-doc
The following NEW packages will be installed:
  cpp dkms fakeroot gcc libfakeroot linux-headers-3.13.0-63
  linux-headers-3.13.0-63-generic linux-headers-generic patch
0 upgraded, 9 newly installed, 0 to remove and 17 not upgraded.
Need to get 0 B/9846 kB of archives.
After this operation, 78.0 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  cpp gcc patch dkms libfakeroot fakeroot linux-headers-3.13.0-63
  linux-headers-3.13.0-63-generic linux-headers-generic

msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"   install 'linux-headers-generic' 'dkms'' failed: E: There are problems and -y was used without --force-yes


FATAL: all hosts have already failed -- aborting

Solution

The solution is to just add the option of force=yes

  
apt: pkg={{item}} 
       state=installed 
       install_recommends=yes 
       update_cache=yes
       force=yes


which is equivalent to what we would have done manually on the terminal

  
sudo apt-get install some-deb -y --force-yes


Every time I use Ansible my admiration for it only increases, how well advanced concepts have been implemented in such simple ways!

Software Defined Environment - Environments on Demand

Get the Development, QA, Staging or Production Environment you need at the click of a button.


Software Defined Environment

The current situation


It wouldn’t be a bold statement to say that all software’s ultimate goal is to enhance the customer experience. How many times have we not read such comments on app stores or heard business say?

 “Great app, but I can only give it three stars until the developers add ...”

But the Development team’s side of the story is

    “I am waiting for the environment to test the code with new features”

Continuous Delivery and Continuous Integration can help release software updates more frequently and with almost no manual intervention, but there are some bottlenecks to being able to do this. Following are a few: -

Delay in getting the Environments


  Lack of self-provisioning creates dependency on IT department.

Lack of easily customizable Environments


For Development, Testing and Staging with new features or updates to dependencies.

Manual Provisioning of Environments


Being repetitive and involving several steps, we would not be able to leverage the power of Automated Deployments and CI.

And the hilarious but unfortunately true risk of

“Oh! But it works on my laptop!”

Not being able to recreate the environments easily and consistently can lead to not being able to recreate performance issues or release code or updates to production confidently!

Inconsistent environments could result in such scenarios as a new update has been released to the production system and the system Admin might have put in the configuration or dependencies that only he or she is aware of to get the app working. Similarly the developer might have put in the unique settings on his or her laptop to get the code working on his or her workstation or laptop. Due to which every server becomes “works of art” and as unique as snowflakes. Needless to say inconsistent environments make it very difficult to determine why an application breaks when it's promoted to the next environment. Wasting the Developer and Operation teams time in determining if an issue is due to the source code or environment configuration.

What is an Environment?


It is not just an image or template of a virtual machine but all the compute, storage, network and several other resources (XaaS) that are required to host your application. Quite simply put, everything you can find inside the server room!

Environments on Demand at the click of a button


A solution that could give the Development, QA, Staging or Production Environment at the click of a button could remove all the bottlenecks and risks that we had discussed earlier and at the same time orchestrate Software-Defined Compute, Networking, Storage, Security and such to provide a smart infrastructure that is aware of resources needed by the application and is adaptive and responsive to the workloads dues to fluctuating business demand. All this while being easy to customize and simple to use!

Please watch this video demonstrating how it works.

How can Software Defined Environments help?


Self-Provisioning, empowers the Developers, QAs and etc. to bring up the environments they need, cutting delay due to dependency and bureaucracy!

Push-button deployments to get environments easily and run automated tests, on any version of the app to any environment, helps in getting faster feedback. Using policies, teams can be allocated quota of resources and authorization to use them can also be fine grained.

Everything can be parameterized due to which getting the n.x update of a dependency into an environment is just a click away.

The other advantages of having a Software Defined Environment is Automating orchestration will vastly reduce the possibility of human error, and make it possible to scale far beyond what people could do manually.

Reducing the cost of cloud ownership by sharing resources, time to market drastically and by being able to reuse the existing hardware resources



Increase the quality of service by improving the application performance by auto scaling which is achieved by having a hardware that is intelligent and adaptive to the needs of the app.

Phoenix Environments

           
Providing resilient, fault tolerant Environments, which can bring up your infrastructure in one click.

Mitigating the risks of inconsistent environments by providing Consistent Environments throughout the software development life cycle, i.e. from a Developers laptop unto the production systems.

Embracing the following advantages of Infrastructure as code, making your infrastructure IMMUTABLE!
           

Extend the advantages of version control from your app to Infrastructure.

Auto deployment will cut the repetitive and manual process of configuring all infrastructure resources.

Get a unified view simplifying the monitoring and management of all resources.

As you test the app at scale and once deployed, there could be hundreds of servers that might need to be brought up at need to scale the app. Using this approach we could cut the repetitive process and copy the configurations to more servers, virtual machines, switches, routers and storage servers instantly.

What do we do with our existing cloud investments?


As we are going to use Open Source Software, like OpenStack, Go CD and such it will be interoperable with any existing private cloud technologies like the VMWare, Xen and such allowing us to reuse the exiting hardware capital and get started with as little investments as possible and scale out by adding in more capacity seamlessly when ever the utilization is expected to reach higher.

Please watch this video demonstrating how it works or let us know and we would be delighted to take you on this journey!

Welcome to the era of Software Defined Economy!



You can see the accompanying slides here. 





The code has been open sourced here. Enjoy!

I will be presenting this in The India Cloud Expo conference

neutron IOError: [Errno 2] No such file or directory: '/proc/sys/net/ipv6/conf/default/disable_ipv6'


Error:
neutron IOError: [Errno 2] No such file or directory: '/proc/sys/net/ipv6/conf/default/disable_ipv6'


Due to this error, the neutron-l3-agent does not start up. End errors out. with the above error. There is a bug filed for this and patched in releases after Icehouse.

Solution

This bug is fixed by doing this

curl -o /usr/lib/python2.6/site-packages/neutron/common/ipv6_utils.phttps://raw.githubusercontent.com/openstack/neutron/stable/icehouse/neutron/common/ipv6_utils.py

Error: Chef::Exceptions::JSON::ParseError: parse error: premature EOF

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
52.0.234.85
52.0.234.85 Starting Chef Client, version 11.18.6
52.0.234.85
52.0.234.85 ================================================================================
52.0.234.85 Chef encountered an error attempting to load the node data for "first1"
52.0.234.85 ================================================================================
52.0.234.85
52.0.234.85 Unexpected Error:
52.0.234.85 -----------------
52.0.234.85 Chef::Exceptions::JSON::ParseError: parse error: premature EOF
52.0.234.85
52.0.234.85                      (right here) ------^
52.0.234.85
52.0.234.85 [2015-02-09T02:36:38-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
52.0.234.85 Chef Client failed. 0 resources updated in 0.854925495 seconds
52.0.234.85 [2015-02-09T02:36:38-05:00] ERROR: parse error: premature EOF
52.0.234.85
52.0.234.85                      (right here) ------^
52.0.234.85
52.0.234.85 [2015-02-09T02:36:38-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
 ✘ ⮀ ~/devops/learnChef ⮀


Solution:


Go to the knife.rb file (generally in ~/.chef/)
And change the


chef_server_url          'https://manage.chef.io/organizations/my_org'

to

chef_server_url          'https://api.opscode.com/organizations/my_org'

error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)

While bootstrapping a redhat node which is on AWS EC2, which supports only the SSH identity file for authentication, I was getting the below error after this command 

knife bootstrap 52.0.13.12 --ssh-user ec2-user --identity-file ~/.ssh/xyz.pem --node-name first1 --run-list 'recipe[some_thing]'


Error

52.0.13.132 Installing Chef Client...
52.0.13.132   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
52.0.13.132                                  Dload  Upload   Total   Spent    Left  Speed
100 18285  100 18285    0     0  48457      0 --:--:-- --:--:-- --:--:-- 48630
52.0.13.132 Downloading Chef 11 for el...
52.0.13.132 downloading https://www.opscode.com/chef/metadata?v=11&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
52.0.13.132   to file /tmp/install.sh.940/metadata.txt
52.0.13.132 trying curl...
52.0.13.132 url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132 md5 b4ccffea24007b83ffdd99b16aea9661
52.0.13.132 sha256 f531541c6786f274bd62fb46bc1ea8f2d70c083e10777b2544c6503c0f90c598
52.0.13.132 yolo true
52.0.13.132 downloaded metadata file looks valid...
52.0.13.132 downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132   to file /tmp/install.sh.940/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132 trying curl...
52.0.13.132 Comparing checksum with sha256sum...
52.0.13.132
52.0.13.132 WARNING: Chef-Client has not been regression tested on this O/S Distribution
52.0.13.132 WARNING: Do not use this configuration for Production Applications.  Use at your own risk.
52.0.13.132
52.0.13.132 Installing Chef 11
52.0.13.132 installing with rpm...
52.0.13.132 warning: /tmp/install.sh.940/chef-11.18.6-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)
52.0.13.132 Installation failed
52.0.13.132 Version: 11
52.0.13.132
52.0.13.132 Please file a Bug Report at https://github.com/opscode/opscode-omnitruck/issues/new
52.0.13.132 Alternatively, feel free to open a Support Ticket at https://www.getchef.com/support/tickets
52.0.13.132 More Chef support resources can be found at https://www.getchef.com/support
52.0.13.132
52.0.13.132 Please include as many details about the problem as possible i.e., how to reproduce
52.0.13.132 the problem (if possible), type of the Operating System and its version, etc.,
52.0.13.132 and any other relevant details that might help us with troubleshooting.
52.0.13.132
52.0.13.132 mkdir: cannot create directory ‘/etc/chef’: Permission denied
52.0.13.132 bash: line 36: /etc/chef/validation.pem: No such file or directory
52.0.13.132 chmod: cannot access ‘/etc/chef/validation.pem’: No such file or directory
52.0.13.132 bash: line 69: /etc/chef/client.rb: No such file or directory
52.0.13.132 bash: line 77: /etc/chef/first-boot.json: No such file or directory
52.0.13.132 Starting first Chef Client run...
52.0.13.132 bash: line 83: chef-client: command not found

Solution

Using the option "--sudo" does the trick



knife bootstrap 52.0.13.132 --ssh-user ec2-user --identity-file ~/.ssh/xyz.pem --sudo --node-name first1 --run-list 'recipe[some_thing]'

52.0.13.132 Installing Chef Client...
52.0.13.132   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
52.0.13.132                                  Dload  Upload   Total   Spent    Left  Speed
100 18285  100 18285    0     0  49365      0 --:--:-- --:--:-- --:--:-- 49552
52.0.13.132 Downloading Chef 11 for el...
52.0.13.132 downloading https://www.opscode.com/chef/metadata?v=11&prerelease=false&nightlies=false&p=el&pv=7&m=x86_64
52.0.13.132   to file /tmp/install.sh.1005/metadata.txt
52.0.13.132 trying curl...
52.0.13.132 url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132 md5 b4ccffea24007b83ffdd99b16aea9661
52.0.13.132 sha256 f531541c6786f274bd62fb46bc1ea8f2d70c083e10777b2544c6503c0f90c598
52.0.13.132 yolo true
52.0.13.132 downloaded metadata file looks valid...
52.0.13.132 downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132   to file /tmp/install.sh.1005/chef-11.18.6-1.el6.x86_64.rpm
52.0.13.132 trying curl...
52.0.13.132 Comparing checksum with sha256sum...
52.0.13.132
52.0.13.132 WARNING: Chef-Client has not been regression tested on this O/S Distribution
52.0.13.132 WARNING: Do not use this configuration for Production Applications.  Use at your own risk.
52.0.13.132
52.0.13.132 Installing Chef 11
52.0.13.132 installing with rpm...
52.0.13.132 warning: /tmp/install.sh.1005/chef-11.18.6-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
52.0.13.132 Preparing...                          ################################# [100%]
52.0.13.132 Updating / installing...
52.0.13.132    1:chef-11.18.6-1.el6               ################################# [100%]
52.0.13.132 Thank you for installing Chef!
52.0.13.132 Starting first Chef Client run...
52.0.13.132 [2015-02-06T07:25:37-05:00] WARN:
52.0.13.132 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
52.0.13.132 SSL validation of HTTPS requests is disabled. HTTPS connections are still
52.0.13.132 encrypted, but chef is not able to detect forged replies or man in the middle
52.0.13.132 attacks.
52.0.13.132
52.0.13.132 To fix this issue add an entry like this to your configuration file:
52.0.13.132
52.0.13.132 ```
52.0.13.132   # Verify all HTTPS connections (recommended)
52.0.13.132   ssl_verify_mode :verify_peer
52.0.13.132
52.0.13.132   # OR, Verify only connections to chef-server
52.0.13.132   verify_api_cert true
52.0.13.132 ```
52.0.13.132
52.0.13.132 To check your SSL configuration, or troubleshoot errors, you can use the
52.0.13.132 `knife ssl check` command like so:
52.0.13.132
52.0.13.132 ```
52.0.13.132   knife ssl check -c /etc/chef/client.rb
52.0.13.132 ```
52.0.13.132
52.0.13.132 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
52.0.13.132
52.0.13.132 Starting Chef Client, version 11.18.6
52.0.13.132 Creating a new client identity for first1 using the validator key.
52.0.13.132 resolving cookbooks for run list: ["learn_chef_httpd"]
52.0.13.132 Synchronizing Cookbooks:
52.0.13.132   - learn_chef_httpd
52.0.13.132 Compiling Cookbooks...
52.0.13.132 Converging 4 resources
52.0.13.132 Recipe: learn_chef_httpd::default
52.0.13.132   * package[httpd] action install
52.0.13.132     - install version 2.4.6-19.el7_0 of package httpd
52.0.13.132   * service[httpd] action start
52.0.13.132     - start service service[httpd]
52.0.13.132   * service[httpd] action enable
52.0.13.132     - enable service service[httpd]
52.0.13.132   * template[/var/www/html/index.html] action create
52.0.13.132     - create new file /var/www/html/index.html
52.0.13.132     - update content in file /var/www/html/index.html from none to ef4ffd
52.0.13.132     --- /var/www/html/index.html 2015-02-06 07:25:47.501523711 -0500
52.0.13.132     +++ /tmp/chef-rendered-template20150206-1075-lczwdi 2015-02-06 07:25:47.502523687 -0500
52.0.13.132     @@ -1 +1,6 @@
52.0.13.132     +<html>
52.0.13.132     +  <body>
52.0.13.132     +    <h1>hello world</h1>
52.0.13.132     +  </body>
52.0.13.132     +</html>
52.0.13.132     - restore selinux security context
52.0.13.132   * service[iptables] action stop (up to date)
52.0.13.132
52.0.13.132 Running handlers:
52.0.13.132 Running handlers complete
52.0.13.132 Chef Client finished, 4/5 resources updated in 10.298957604 seconds