ENC & Hiera

Hiera is included as part of puppet 3.0, but if you’re running puppet 2.7 you’ll need to install the following packages (for debian): hiera, hiera-puppet

 

ENC

  • Executable with one parameter (fqdn)
  • Flexible – you design how the information lookup is done (query a database, parse a hostname or other Facter fact, etc).
    • Can be written in any language: shell, perl, ruby, python, etc…
  • Can get info from filesystem: cat $1.yaml
  • Get info from inventory database
    • Plugs into your existing CMDB (Configuration Management Database) to retrieve information that already exists in another source of truth

 

Puppet comes with an ENC (Graphical UI for managing nodes).

  • Supports setting parameters
    • Create groups, set parameters on those groups, assign nodes to a group, and then access those parameters as variable in the code base.
  • Set the node environment
  • Set top-scope variables
  • Easily see every class/param applied to my node in one place.
  • I can change configuration data in my ENC via its API, from Jenkins or
    arbitrary cli scripts…

The code to lookup a param set in the ENC is…

lookupvar(“::#{module_name}_#{var_name}”)

CODE

/etc/puppet/puppet.conf

[master]

node_terminus = exec

external_nodes = /etc/puppet/bin/my_great_enc.sh

 

/etc/puppet/bin/my_great_enc.sh www1.domain.tld

parameters:

location: de-ber2

classes:

ntp:

ntpserver: 10.2.2.2

apache:

mysql:

environment: production

 

Hiera

  • Hierarchical data structure
    • Hiera iterates over it hierarchy lists in order looking for a matching file and then for a matching param until the one you are requesting is found.
  • The main benefit would be having actual data files that we can access from the Puppet manifests.
  • Gives us the ability put data in a way that maps well to the natural structure of compute environments.
  • Data is truly separated from your Puppet code—it exists in an entirely separate directory structure.
    • Supports structured data—like arrays and hashes—
  • Custom backend that talks to your DB that has a web frontend that administers the data in question.

 

CODE

/etc/puppet/hiera.yaml

:hierarchy:

- %{operatingsystem}

- common

- %{datacenter}

- %{serverfunction}

:backends:

- yaml

:yaml:

:datadir: ‘/etc/puppet/hieradata’

DATA

/etc/puppet/hieradata/Debian.yaml

ssh_packages: openssh-server, openssh-client, openssh-blacklist

/etc/puppet/hieradata/CentOS.yaml

ssh_packages: openssh, openssh-clients, openssh-server

USAGE

/etc/puppet/modules/ssh/manifests/init.pp

class ssh ( $ssh_packages = ‘default’ ) { <– Puppet 3.x

$ssh_packages = hiera(‘ssh_packages’) <– Puppet 2.7.x

package { “${ssh_packages}”: ensure => present }

}

Hiera was not intended to be an ENC alternative or replacement. It did replace extlookup, which in turn was kind of a crutch for people who didn’t want to invest the effort of creating a full fledged ENC.
Hiera should not be a reason to deprecate the ENC concept altogether.

Hiera saves 80% of us from writing their own ENC (deceptively complex process) then it’s been a success, but it’s aim is not to replace all ENCs.

 

Pull Request made simple.

So you made some cool changes to an open source repo but is your first time making a pull request? … (All commands below)

Is not that hard! Here’s what you need:

Maybe you’ve already done this (Do it if you haven’t) :

  • Fork the Open Source repo.

  • Create a branch (On your forked repo). On it will be the changes you made. Lets name the branch first_contribution

Commit everything to your branch. Make sure everything works with your changes.

That’s basically it!

To finish:

  • Click the “Pull Request” button.

  • Give a subject to your request.

  • Give an explanation of what you made.

  • Wait ‘til you get feedback! … And make the process again.

It’s organized to use branches for issues. Make them ready and then merge all in your master.

Here are the commands you may need.

  1. Create branch:
  2. move to that branch:
  3. Push to your branch

$ git checkout -b first_contribution
$ git checkout first_contribution
$ git push -u origin first_contribution

 

Easy? Short? It is.

Here’s the GitHub Guide

And remember:

Command line VS. Puppet basic comparison

So you use the command line and every time you set a developer environment use the same commands over and over? … Come on… XXI Century! .. Puppet is made for that and you’ll see here what magic can you do really easy…

In this tutorial we’ll see how much alike is puppet with the command line and how can you automate some of the more common tasks.

Every time we download files, create files… add some permissions to them.. execute them… a new user.. a new group… install services right? .. an every day task. Here are some comparissons:

Files

On command line: new file, new permissions, execute:

touch /home/user/programs/myProgram.py
chmod 755 /home/user/programs/myProgram.py
./home/user/programs/myProgram.py

That every time for the same Program? … This is a quick comparison of the same commands on Puppet:

file { "/home/user/programs/myProgram.py":
ensure => file,
mode => '755',
owner => user,
}

What are we doing there? … Set where the file should be. Ensure it is a file. (It could be a directory also) Give it permissions and owner.

You can also determine a content of a file… this will come later…

Commands

You download the .py instead creating it? The same on /home/user/programs/

wget https://go.googlecode.com/files/setup.py

On puppet it could be:

exec { "Download_installer":
command => "/usr/bin/wget https://go.googlecode.com/files/setup.py,
cwd => "/home/user/programs/",
creates => "/home/user/goLang/setup.py",
}

What are we doing? Set the name of the function “Download_installer”, specify what command we are using, specify where is the path to execute the command, and tell what will be created.

Services:

This is actually very simple… always installing nginx? (on ubuntu for example)

sudo apt-get install nginx

So in puppet:

package { "nginx" :
ensure => present,
}

that was easy…

This is just the beginning… Start making your own recipes and start having an easier life from now on….

Puppet can be used in a level where everything works by itself, but everything starts with your first recipe. Dig into it. We’ll add more tips and an easy way to begin everything… Your imaginations is the limit…

Coming next: Automated recipe to set up your GoLang development environment.

The coolest Emacs in 4 steps

Emacs like a Sir.

—-

Every EMACS user knows how easy your life gets when the “learning-Emacs-Fase” ends, for which I recommend this command sheet.

___

Along with the #RTFM (Read The Farguin Manual) step. Which you can read when you install emacs and open it.

Any way. Here’s a 4 step guide to enjoy a really cool EMACS installation with some features you surely will enjoy

1. Install EMACS24

 

First of all, add the repositories:

sudo add-apt-repository ppa:cassou/emacs
sudo apt-get update

sudo apt-get install emacs24

Why emacs24? .. for the further steps you may encounter less problems if you do so but any version of emacs you have works… Personally I’ve just tried this steps with 23 and 24 version

2. Prelude

 

Have you Emacs guys heard of this?… Well, with prelude into your emacs you get the possibility of enhance the way you work. Git required.

wget –no-check-certificate https://github.com/bbatsov/prelude/raw/master/utils/installer.sh -O – | sh

You will get stuff like:

  • Command completion,
  • Other languages mode (php, python, etc…),
  • Color Themes to be more confortable.
  • … A lot of more stuff.

3. Know your new prelude

https://github.com/bbatsov/prelude

of course, on Git.

There you should check:

  • how to enable Additonal Modules,
  • Keymap,
  • manage color themes, and more.

4. Tips

 

First of all. Emacs exist to be used on terminal (<3), so you should know some things that I’ve found:

  • Add to your .bashrc file:

export TERM=xterm-256color

With this you’ll avoid any color problem.

If you want to edit any theme, the path of these is:

.emacs.d/elpa/

there you can modify color themes and more to have a more confortable experience.

and to enable modules:

/.emacs.d/modules

and you activate them on

/.emacs.d/

just be sure to have what you import. As Always.

Also you could check this question on stackOverflow for any color problem:

http://stackoverflow.com/questions/17842144/emacs-colors-why-it-is-gray-on-current-line-zenburn-theme/17847682#17847682

Enjoy.

copy and paste from emacs

So we’re using emacs for almost everything, and we’re getting used to do everything with it, but we faced a problem we think everyone who uses emacs has have, we mean, copy/paste from emacs to X-windows, so, after overcome fear to make changes to emacs, and a little google we found this solution:
 
First of all we need to instal xsel:
 

sudo apt-get install xsel

 
after that we need add this to the end of .emacs file:
 

(setq x-select-enable-clipboard t)
(unless window-system
(when (getenv "DISPLAY")
;; Callback for when user cuts
(defun xsel-cut-function (text &optional push)
;;Insert text to temp-buffer, and "send" content to xsel stdin
    (with-temp-buffer
     (insert text)
     ;; I prefer using the "clipboard" selection (the one the
     ;; typically is used by c-c/c-v) before the primary selection
     ;; (that uses mouse-select/middle-button-click)
     (call-process-region (point-min) (point-max) "xsel" nil 0 nil "--clipboard" "--input")))
 ;; Call back for when user pastes
 (defun xsel-paste-function()
    ;; Find out what is current selection by xsel. If it is different
    ;; from the top of the kill-ring (car kill-ring), then return
    ;; it. Else, nil is returned, so whatever is in the top of the
    ;; kill-ring will be used.
    (let ((xsel-output (shell-command-to-string "xsel --clipboard --output")))
     (unless (string= (car kill-ring) xsel-output)
   xsel-output )))
 ;; Attach callbacks to hooks
 (setq interprogram-cut-function 'xsel-cut-function)
 (setq interprogram-paste-function 'xsel-paste-function)
 ;; Idea from
 ;; http://shreevatsa.wordpress.com/2006/10/22/emacs-copypaste-and-x/
 ;; http://www.mail-archive.com/help-gnu-emacs@gnu.org/msg03577.html
))

If you don’t want to restart emacs, you can open .emacs if not, and select the code above and then with

M-x eval-region

you’ll have it set and running.

Source:

http://hugoheden.wordpress.com/2009/03/08/copypaste-with-emacs-in-terminal/

RabbitMQ

We’re doing a lot of work and putting on it a lot of effort, since we don’t know how to do practically anything, we have to study a lot, it’s not we know nothing, it’s just things we need to make our project are new for us, things like GuruPlug, CUPS or even the VPN are easy now, but not at the beginning and it’s the same for programming, python is what we’re using, it’s very friendly and we can do a lot with a few lines… one of those things we needed and had to learn was RabbitMQ:

 

 

 

 

 


 
RabbitMQ is an Advanced Message Queuing Protocol (AMQP), which allows to control, in very easy way, the way we control messages throw our architecture, cause when you do the tutorials they provide, you have almost everything you need to accomplish the goal, so we need just some hours and imagination to make it works… so basically what we did was to get the code of the last of the tutorials and modify it…
 
It uses something called Remote Procedure Call which allows clients to call a function defined in the server side, it’s something like “Hey dude, let’s make some noise!!” and the “dude” starts to do what he has to, in this case, the “Consumer” is in the GuruPlug device and the “Producer” is in the Queue Server side, according to our architecture, it’ll work throw VPN and became an excellent option because the queue will wait until an answer is provided by the consumer.
 
The code of this part of the project will be on our BitBucket, where we’ll store all related to the project but I’ll do a little explanation about this one…
 
We’re using pika, which is the python implementation of RabbitMQ… so, first of all we need to create a connection in both consumer and producer:
 


connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))

channel = connection.channel()

channel.queue_declare(queue='rpc_queue')

 
After this you have to define what is going to be called and start receiving petitions:
 


def on_request(ch, method, props, body):
    archivo = body
    print " [.] TU archivo: (%s)"  % (archivo,)
    response = func_imprimir(archivo)

    ch.basic_publish(exchange='',
                     routing_key=props.reply_to,
                     properties=pika.BasicProperties(correlation_id = \
                                                     props.correlation_id),
                     body=str(response))
    ch.basic_ack(delivery_tag = method.delivery_tag)

channel.basic_qos(prefetch_count=1)
channel.basic_consume(on_request, queue='rpc_queue')

print " [x] Awaiting RPC requests"
channel.start_consuming()

 
This is the code used in the consumer, we don’t post the function “func_imprimir()” but if you want to chek it out you could do it in our repository

Well this is all we did in the consumer’s side, in the producer’s side we have this code:
 

#!/usr/bin/env python
import pika
import uuid
import os
import sys

class FibonacciRpcClient(object):
    def __init__(self):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
        self.channel = self.connection.channel()

        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)

#Se define la respueta que se manda al server
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    #def call(self, n):
    def call(self, archivo):
#        print "Entre al call. este es el archivo:" + archivo
        self.response = None
        self.corr_id = str(uuid.uuid4())
        self.channel.basic_publish(exchange='',
                                   routing_key='rpc_queue',
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ), body=archivo.read())
                                  #body=str(n))
        while self.response is None:
            self.connection.process_data_events()
        #return int(self.response)

fibonacci_rpc = FibonacciRpcClient()

print " [x] Requesting fib(30)"
archivoAImprimir = open(sys.argv[1], 'rb')

response = fibonacci_rpc.call(archivoAImprimir)

archivoAImprimir.close()
print " [.] Got %r" % (response,)

 
As we can see, first we open the communication, after that, in the function called “response” we just check that the id petition match with the one we’re looking for, remember we’re going to print a lot of documetns and there’s no place for mistakes… and later on we use the function “call” where we are going to call, precisely, the function on the consumer… and that’s it, it works, we just run the consumer in order to wait for requests, and the producer with the path to the file we want to print :) if you have any doubt you could ask us in the comment section….

And remember… #RTFM =D

Creating your RFS UBS Stick =)

Driver USBSo… At this point, you must be very happy with you #GuruPlug because it’s not being confusing at all configuring it to make anything you want… (Yeeeeaahh… sure… ) .. haha.. we know.. It is really cool work, but must maintain you mind fresh.. relaxed… and keep it from blowing…

Anyway, here are the steps we followed to create an excelent working USB to boot in you fully functional GuruPlug. Yup, this is a RFS(Root File System) USB, so actually you are working into the USB, not the GP.

Check the Formatting your USB using Fdisk post so you can do all this with no problems.

Here’s what we’re going to do here:

1 format the USB
2. Add to it the Kernel Partition
3. Add to it the RFS Partition
4. Insert into GP.
5. Be happy…

…. yup… it sounds easy….. (so enjoy the “sound”) … Fresh formatted usb? .. lets begin because we need 2 partitions there. Watch this:

Type fdisk /dev/sdbX (x is your usb number[scan with “df”])
If it is the only usb you are using, then type Type fdisk /dev/sdb and then, when you open fdisk remember to type “d” to erase the partition.

FIRST PARTITION

Type n (new partition on your USB)

Select your Partition type:
(from here you can type just enter..)
Type p (primary partition)
Type 1 (Partition number)
Type enter (Use default)
Type +100M (To make this 100Mb size partition)

SECOND PARTITION

Type n (new partition on your USB)

Select your Partition type:
(from here you can type just enter..)
Type p (primary partition)
Type 2 (Partition number)
Type enter (Use default)
Type enter (Use default size partition.. the rest of it)

SPECIFY THE PARTITION ID FOR EACH PARTITION

Command (m for help): t
Partition Number (1-4): 1
Hexadecimal code (Type L to view codes): 6 //fat16

Command (m for help): t
Partition Number (1-4): 2
Hexadecimal code (Type L to view codes): 83 //Linux

Command (m for help): w (write all changes)
Partition table modified!

After this format each partition:

mkfs.vfat -F 16 -n dp_kernel /dev/sdb1

mkfs.ext3 -b 4096 -L dp_rfs /dev/sdb2

So. It’d look like this on GParted:

rfsgparted

Great! .. Almost there…
Here are the downloads for the GuruPlug.

All
UImage(Kernel)
RFS

Make sure you have all this.

Finish our USB

Untar and copy the uImage and rootfs.tar.bz2 to this newly prepared USB
stick.

- Plug the USB stick to the Host and execute the following command as root user in the Host Terminal.
The example here shows the uImage(kernel image) and rootfs.tar.bz2 have
been downloaded to /home folder in the Host, and the two partitions /dev/sda1 and /dev/sda2 in this newly prepared USB stick are mounted to /media/usb0 and /media/usb1 in Host
respectively.

usb0 -> dp_kernel
usb1 -> dp_rfs

#cd /home
#cp uImage /media/usb0
#tar xvf file.tar.bz2 --strip-components 1 /media/usb1

you should use this line because on the RTF part must be all the root directories. So this will extract skipping the first folder that has the RFS


….

#cp uImage /media/usb1/home
#cp rootfs.tar.bz2 /media/usb1/home
...(we will use the kernel image and file system later)
#sync
#umount /media/usb0
#umount /media/usb1

At the end of this you should have on the kernel partition the uImage and on the other partition the root file system. This is very important or you’ll get the next error:

Kernel panic - not syncing: No init found

Partitions must be like this:

1. fat16 extension . Here goes the uImage just like that.
2. ext3 extension . RFS. (Direct the /home, /etc, /var…)

Here’s a nice RFS image:

rfs

IMPORTANT. DO NOT UPGRADE TO SQUEEZE YET…

We should keep the Debian version (Debian 5.0 – Lenny), if you want to change to squeeze, there would be no problem, updating the sources.list to squeeze, however if you get to turn off the guru, when you want to turn on again, we will see an error like this:

attempt to access beyond end of device
sdb: rw=0, want=3636361172, limit=398297088

This error occurs because the kernel (in fact for lenny in the guru) search for stuff that are in lenny but not in other versions, or it just simply can’t find the path to boot from the usb… so this error causes an infinite loop .. This also happens if you upgrade all with no remorse… you’ll have a great squeeze… until you restart the gurú… so… do not upgrade.. keep reading..

remember that changing the marvell on the guru (post) usually, the external USB stick is always recognized as /dev/sdc* , while the internal boot uSD card is /dev/sda* and in our case it was /dev/sdb .. so .. make sure that everything is in the right place…

The way I solved it was remaking the bootable usb for guru, because this causes an error in the partitions.

Once created it all again, I could ssh access to the device, in which you must follow the next steps:

apt-get update

we’ll get some “No more security actualizations for lenny” warnings .. so we must keep up to date this parts manually…

Now we are ready to

apt-get upgrade.

Pay attention to this, because you’ll get some warnings of which file you want to keep… Always keep the one that comes with the guru. ALWAYS!

After that, you can do watever you want to your RFS….

In our case we got openvpn & cups, also (Obviously..) VIM

Hope everything work well to you… any problems, contact us! =)

Check this GREAT documentation of the dream-Plug…
DreamPlug – Change OS from Ubuntu to Debian-20110617
Will make your day in case you are having troubles with this…

Formating USB using Fdisk (NO GUI)

If you want to make a simple quick partition and you have no GUI, here’s what you should do. (Besides #RTFM)

We’ll try to make this simple. To the point. If you need a quick format, then you won’t like reading a lot.
Follow this steps:

You can use the df command to scann your mounted devices

Open a terminal and type sudo su
Type fdisk -l (and note which device is your USB)

# fdisk /dev/sdbX (X is the number of your usb device)

Now, you enter into a menu where you can press ”m” to get help and see all the stuff you can do. But to our purposes we’ll:

1. MAKE YOUR USB A SINGLE PARTITION

Type d (To Delete the existing partition)
Type 1 (to select which partition will be deleted)

Now, you’ll see something like this: (steps below)

fdisk1

Type n (new partition on your USB)

Select your Partition type:
(from here you can type just enter..)
Type p (primary partition)
Type 1 (Partition number)
Type enter (Use default)
Now you decide the size of the partition. If you want to use it all, use default.
Type enter (Default [the whole disk])

Now change the partition type id.

Type t (choose type)
Type 1 (choose your partition number)
Type 83 (Linux system id ext3) [is the most common, but you may type “L” to see all the types

At the end. If everything went well, You should type:

Type w (To write)

After this, all your changes will be done.

Well.. after all the complicated stuff, you can use simply these next lines to format partitions:

in the /dev/ path are all the devices. Scan typing “df

Unmount the device you want to format.

umount /dev/sdb1

Fat16:
(-n is used to label the formated disk)

mkfs.vfat -F 16 -n MyUSB /dev/sdb1

Ext3:
(-L is the label)
(-b size per block. Can take 1024, 2048, 4096… )

mkfs.ext3 -b 4096 -L MyUSB /dev/sdb2

NTFS:

mkfs.ntfs -v -L data /dev/sdc1.

there are also this other types of quick format:

mkfs.btrfs
mkfs.ext2
mkfs.ext4
mkfs.msdos
mkfs.vfat
mkfs.cramfs
mkfs.ext3
mkfs.ext4dev
mkfs.ntfs
mkfs.xfs

Well. Connect and disconnect.. and… Enjoy. =)

Connect To Gurú Using JTAG

All right! You have your brand new GurúPlug and your computer…. mmm now what? .. here Goes the recipe! … Let those ingredients make your day:

  • 1 Gurú plug
  • 1 JTag (should be buyed with the gurú)
  • 1 Mini-Usb Cable (included with the Jtag)
  • 1 Minicom Installation
  • 1 Ac/Dc Cable to power up the Gurú
  • 1 LA. Woman (recommended)
  • 1 pair of headphones (recommended)
  • 1 M&M’s Pack (optional)
  • 1 Bottle full of water (optional)

I’ll describe what to do with the not so obvious Ingredients. Ok? .. haha let’s begin..

To connect, first of all you need to install minicom. You know the syntax of your Linux distro… for example using ubuntu it would be:

sudo apt-get install minicom

Great! .. now, we need to open it.

sudo minicom -s

It’ll display a menu of options. Here, we need to configure everything so the gurú and the minicom can comunicate.

# minicom –s

Set the Configure properties as follows:

Bits per sec field to 115200
Data bits to 8
Parity to None
Stop bit to 1
Flow Control to None.

What do I Mean? .. Here:

.

(The USB1 part may be different depending on the port you connect the JTag. Check troubleshoot at the end)

.

After doing that you could “Save setup as…” and name it ‘marvell’… or however you like so the next time you need it.. .(And you will) .. you can run it as:

(don’t do it yet)

$ sudo minicom –o marvell

Now! .. we need to connect everything. Like this:

and the JTag must be like this:

Don’t forget! the jumper wire on the UART selection!.

Don’t connect the JTag to the PC yet…

….wait…. Now connect it. (GuruPlug is off)

Open terminal, and run the configuration we saved before.

$sudo minicom -o marvell

you should see the following output:

AT S7=45 S0=0 L1 V1 X4 &c1 E1 Q0

that means it is ready to enable the connection.
(If you’re having troubles… check the Troubleshoot part)

NOW PAY ATTENTION. TRICKY PART

Now that the config is running and you got those alphanumeric symbols, we connect the guru to power and start pressing “enter” a couple of times to stop the autoboot and have access to marvell. It sometimes doesn’t show a thing until we stop the autoboot and access, that’s why you should press enter … if it didn’t work after a few seconds.. you are not doing it ok.. try again disconnecting the Guru plug, and make the troubleshoot part.

If you don’t do this, you’ll access your gurú the same way ssh does.. If this is what you wanted to do, then you did it with the complicated way.. anyway.. it will ask for the Login and pass for the gurú

Login : root
Password: nosoup4u

If you actually got the marvell connection, then congrats!! .. Complementi! .. now: RUN THIS COMMAND: the command printenv so you can see what its going on with your gurú config and back it up!!! it will save you a lot of times.

We finished! .. now continue making your own USB bootable GurúPlug. =) … So you can make any mess you want.

.

TROUBLESHOOT

This will be an actually small troubleshooting. Because, there are not a lot of different tries of what you should do…

First. What could happen is that you are not specifying the right port. In the image it says “Port /dev/ttyUSB1”. With me it worked using “Port /dev/ttyUSB0”.

Next, What also worked for us, is:
press ctrl + A , then press “Z” so the options come up, there we press “M” to connect.
(With H you hang up). Then it should show:

IF you want to close the mincom connection, use Ctrl A,+ Q

AT S7=45 S0=0 L1 V1 X4 &c1 E1 Q0
And now we proceed connecting.

Check this link! Here’s everything! The Quick Start Guide for the plugs.

if you’re doing more complicated things then you’ll like this: