Providing Solutions For Life

Unix cheat sheet

Inspite of years of working on Unix and Linux I still wasn't confident if I knew enough and hence read it in these holidays and I see that we can get our way through these amazing OS using the below commands:-

The OS uses the Filesystem to store data on the harddisk. They are different types of file storage, NTFS, FAT and etc. 

Commands to get your through!

pwd ==> Prints the working directory you are in. 


ls ==> just to list the files and directories, flags
  • -l  => output in list format
  • -a => show the hidden files
  • -h => output is more human, that is easier to understand 
cd ==> Change directory 

find location -type f -name filename   ==> to find the file that you have been looking for.

parted for partitioning ext4

mkfs ==> to create filesystems

echo $PATH  ==> print the path

env ==> to list the environment variables of 

ps -efl  ==> to print the details of the process that are running

ps $$ ==> to get PID of current shell

jobs ==> List background processes

!! ==> to run the last run command

history ==> very important to see what all commands have been run.

sleep 30 ==> to lock the terminal from taking any input for a certain amount of time.

whoami  ==> if you suffer from short term memory loss :) this helps you to know what is the log in you are using currently. 

nohup ==> prefixing it in front of a command makes the process to be immune to any futher signals until the completion of the task. 

& ==> appending this to a command runs it in the background.

fs <job_number> ==> brings the process to foreground.


Global profile (/etc/profile)
User profile (~/.bash_profile)



[million $ question is what is the difference between a job and a process? 
Jobs are one or more processes that are grouped together as a job, where job is a UNIX shell concept.]

Accessing the Google APIs using Oauth 2 in Python

In its simplest form, Oauth is as in this document I had created earlier.

Oauth is such a simple protocol and so much of fuss is being made about it, its so simple that you can understand it even by reading its Specification documentation!

Below pic shows the flow

     +--------+                               +---------------+
     |        |--(A)- Authorization Request ->|   Resource    |
     |        |                               |     Owner     |
     |        |<-(B)-- Authorization Grant ---|               |
     |        |                               +---------------+
     |        |
     |        |                               +---------------+
     |        |--(C)-- Authorization Grant -->| Authorization |
     | Client |                               |     Server    |
     |        |<-(D)----- Access Token -------|               |
     |        |                               +---------------+
     |        |
     |        |                               +---------------+
     |        |--(E)----- Access Token ------>|    Resource   |
     |        |                               |     Server    |
     |        |<-(F)--- Protected Resource ---|               |
     +--------+                               +---------------+



Authentication and Authorisation are two different things, here we will cover authorisation.
Let's take Python 

Now the first thing is to get a consumer key and secret. Which can be go by going and registering on 
on the API Access pane of the Google APIs Console.



Google recommends that we use client libraries as they are safe hence we will be using the oauth2client-module. Which is already there within the Google's Client Library.

Before we go ahead, lets discuss the "flow"

First using the Flow we will get credentials which can access a certain service and that will be stored. 

FLOW ===> Credentials ==> Storage

"Flow" class -- flow_from_clientsecrets 


got by from oauth2client.client import flow_from_clientsecrets
This is used to acquire credentials that authorise our app (to access user's data). In order for a user to grant access, OAuth 2.0 steps require your application to potentially redirect their browser multiple times. Lets see below how the flow object is instantiated using the flow class.

flow = flow_from_clientsecrets('path of client_secrets.json',
                        scope='https://www.googleapis.com/auth/Resource',
                               redirect_uri='http://example.com/auth_return')
you can download a client_secrets.json file from Google API console for your app and place it in the folder where the python script is located.
Scope is telling the script that you want access only a particular resource, like Google Groups and calendar. This is like fool proofing in case you loose control of this script, it could go crazy on all the resources. So we limit it upfront-- Damage control ;)

Though as per the flow, Credentials should have been the next topic, we just have a place holder for the credentials to be defined. 

Storage 

First we import this class by  
from oauth2client.file import Storage
observe that its oauth2client.file and not client
then 

storage = Storage('some file.dat')
this command will create a file by that name, and have the credentials stored into it for further transactions.

Now for the credentials finally, but before we go there, oauth2client.tools.run() from its documentation here is what it says

The run() function is called from your application and runs through all the
steps to obtain credentials. It takes a Flow argument and attempts to open an
authorization server page in the user's default web browser. The server asks
the user to grant your application access to the user's data. If the user
grants access, the run() function returns new credentials. The new credentials
are also stored in the Storage argument, which updates the file associated
with the Storage object.

so to get it into python we do 

from oauth2client.tools import run

Credentials 

The Flow object can create the Credentials for you.  
A Credentials object holds refresh and access tokens that authorize access to a single user's data. 
so finally to create credentials

credentials = run(flow, storage)

So the now the bare minimum code so far is as in the screenshot below, so when we run it, it opens our default browser as below


A file which we had specified earlier called "some file.dat" gets created and holds all authorisation token and etc now, which has the info as below

{"_module": "oauth2client.client", "token_expiry": "2013-11-29T15:13:12Z", "access_token": "y"...}

Now to avoid having to go through the process of going to a browser and authorising this app always, what we can do is check if there are already some credentials in the stored file and if they are there use, them else create a new one. 

if credentials is None or credentials.invalid:
        print 'invalid credentials'
        # Save the credentials in storage to be used in subsequent runs.
        credentials = run(FLOW, storage)


Finally, using these credentials we have got!

These objects are applied to httplib2.Http objects to authorize access. 

We will use the authorise() function of the Credentials class to apply necessary credential headers to all requests made by an httplib2.Http instance:

httpplib2.http is used to place requests to the web, see the below screenshot, where I am just trying to get the unauthenticated page, like Google. 

from httplib2 import Http 
note the caps in Http



so we authorise our request using the "credentials' function:

http = credentials.authorize(http='h')

Once an httplib2.Http object has been authorised, it is typically passed to the build function:


service = build('calendar', 'v3', http=http)

service = build('groupssettings', 'v1', http=http)

Agile Python Development and the Laundry list

We need method to the madness!

There are apparently two, one is waterfall and the other agile

When?

Manifesto for Agile Software Development was produced in February, 2001.

Why Agile?

Is more pragmatic, as it knows that not everything can be documented at the beginning of a project. But it does put in place some rigorous  checks to accommodate these changes!

So that any change can be accommodated, with good communication, iterative design and development. 

Agile development can be applied in any project, both testing tools and build automation tend to be very language specific. Python too has these tools.

Jargons again!

Agile methodologies 

  • XP - Extreme programming -- focuses almost exclusively on the developer and development techniques
  • DSDM - Dynamic Systems Development Method -- focuses completely on processes
  • there could be many more.

Spirit of it from manifesto!


Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan


Agile methods has collection of various techniques:-

Laundry list


on-site customers - person using the product.

pair programming - being a solo role since the beginning of my career I know how it feels to be banging your head alone to a computer, its better for two to be doing this.

sys metaphor - using the same names all around the program and treating anything else as bugs, IDEs refactoring tools help in changing names across the system.

IDEs are integrated development environments, like Pycham, Eclipse and etc. 

Refactoring is the practice of simplifying and clarifying code when ever possible.
 
Documentation - yes you read it right! there is documentation too but it should be minimal. It should be limited to that which is necessary to ensure that participants in the development process can communicate.

Simple Design is fulfilling requirements of the user stories and nothing more. the code is the design, should be as simple as possible.

Principle for it:-
- DRY - Don't repeat yourself, when you see the same thing being done again and again in your code, refactor! May be create a method or something also called as DIE - duplication is evil.  
- YAGNI - "You aren't gonna need it" -- "do the simplest thing that could possibly work" (DTSTTCPW) no point in carrying the useless burden around.
- KISS: is an acronym for the design principle "Keep it simple, Stupid!".
Better raw than wrong - just say things in simple and direct way. 

Short Iterations - The customers and the dev team meets on regular intervals, generally every week and discussion on what has been done, give a demo may be and then plan for what needs to be done in the coming week. This is to avoid any tragic surprises to the client. 

Collective Code Ownership -  it's not one man or his fathers property, due to pair programming, the strong attachment to the code is dissolved.

Producing short iterations is made possible through automation. There is automation from the Developers desk to the production environment. Human touch is only needed when things go wrong!

It is human to err and hence manual process should be avoided. 

Continuous reflection:  Is ongoing analysis of how development is going on.
documenting everything that is being done can CYA! Trust me, I've realised this in the hard way, so no matter which company you are in keep shouting and bragging on what you are doing and if possible give them some jazzy acronymos or create a buzz word so that others are intimidated on hearing it. This is how this God forsaken industry operates unfortunately. 

Continuous Integration:

Have you saved the file, or do you have a back up, if this words make your heart skip a beat then you should remember to check in the code into the repo as often as possible as a single place keep all the artifacts necessary to produce the build.
This could include build scripts, tool scripts, properties files, installation scripts, third-party libraries, tests, and tool configurations (e.g., IDE configuration files).

Since the program is built, installed, executed, and tested throughout development. This guarantees that there will be no deaths in the team at the time of production!


Test Driven Development:

Tests are written before the actual code is written to solve the problem. 

Types of tests are

Unit tests or programmer tests is level of methods and functions. With this tests changes are easy to make as they detect the bugs immediately. Unit tests fall into two broad categories: programmer tests and customer tests. What they test distinguishes them from each other.
  • Programmer tests prove that the code does what the programmer expects it to do. They verify that the code works. 

  • Customer tests (a.k.a. acceptance tests) help to ascertain that the code behaves the way the customer would expect it to. It verifies behaviour at the level of classes and complete interfaces. 
Unit testing is a part of regression testing.  
Regression tests identify bugs that have been seen before and that had been fixed. Unit tests should run fast else, programmers will not run them and that could lead to a lot of bugs at the end. 
Two of the most common tools in Python for unit testing are unittest and Nose.

Functional testing verifies that the complete application works as expected. Functional testing is done by QA department who are directly integrated into the development process. It verifies that customer sees what they would expect to see and no bugs have crept in. 

I still haven't got Agile, what the heck is it?

It simply means that you will have the requested app or program at the earliest even though it might not have the full functionality of the finished application!
 

Google APIs for python - don't feel unfortunate anymore if you have to work on these

From the past 4 years I've been unfortunate enough to work on Google API's which are those for which even Google cannot search their documentation, may be because it doesn't exist, stackoverflow no matter how much it overflows cannot give us anything much on these.

Below is the summary of all my experience to crack the Google APIs.

I've spent days together to crack one thing at times and I wished I had known these tricks so that it wouldn't have taken me this long.

So firstly try to read as much as possible from the getting started guide. This will take you to a world of links, go to those link and grab in what ever is possible (don't worry if you don't understand anything). [trade secret of software engineers is make the simple sound as complex as possible, and intimidate others!]

But the world of python is different! 
"Simple is better than complex." -- zen of python

So the magical commands in python are :-

dir()
type()
help() (keeping the best for the last part)

also inspect.getargspec()

Lets discover each one of them:-

dir() - To list the names of built-in functions and variables. 

(though it doesn't give a complete list. If you want a list of those use the standard module __builtin__ eg. dir(__builtin__)
For example if you have a class and instantiated an object for eg.
service = discovery.build('admin', 'reports_v1', http=http)
and you want to know what are the functions this object can give us.
just run dir(service) to get the list as below, (ignore the u' they are the UTF encoding stuff)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_basic_methods', '_add_nested_resources', '_add_next_methods', '_baseUrl', '_developerKey', '_dynamic_attrs', '_http', '_model', '_requestBuilder', '_resourceDesc', '_rootDesc', '_schema', '_set_dynamic_attr', '_set_service_methods', u'activities', u'customerUsageReports', u'userUsageReport']
so we can see that the above result there could be 3 methods. Now to know what methods are there inside these methods, we can do this dir(service.customerUsageReports()) and get a list
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_basic_methods', '_add_nested_resources', '_add_next_methods', '_baseUrl', '_developerKey', '_dynamic_attrs', '_http', '_model', '_requestBuilder', '_resourceDesc', '_rootDesc', '_schema', '_set_dynamic_attr', '_set_service_methods', u'get', u'get_next']
so we come to see that there are two more methods get and get_next for us to use!


Now if at some point you would like to know what the datatype of the object is use 

type(service.customerUsageReports()



this will give the type of the object eg. <class 'apiclient.discovery.Resource'>
Now for the best one help(), my fascination for python has increased all the more after knowing about this command. It's awesome. See how it can help us
Now how do we know in the really mediocre documented Google world, what are the parameters "get method" takes


help(collections.get)


gives the below
method(self, **kwargs) method of apiclient.discovery.Resource instance    Retrieves a report which is a collection of properties / statistics for a specific customer.        Args:      date: string, Represents the date in yyyy-mm-dd format for which the data is to be fetched. (required)      pageToken: string, Token to specify next page.      parameters: string, Represents the application name, parameter name pairs to fetch in csv as app_name1:param_name1, app_name2:param_name2.        Returns:      An object of the form:            { # JSON template for a collection of usage reports.        "nextPageToken": "A String", # Token for retrieving the next page        "kind": "admin#reports#usageReports", # Th

seeing this we know that it takes the date parameter in a certain string format. This saves lives! :)


Adding to the above, we see the below which could also be used and might help, though I haven't been much helped by this

 import inspect


inspect.getargspec(someMethod)
Get the names and default values of a Python function’s arguments. A tuple of four things is returned: (args, varargs, keywords, defaults). args is a list of the argument names (it may contain nested lists).

No more head breaking folks!  

instead break the code! 

Installing Chef client in less than a minute

Open a terminal

Enter command as in the screenshot below, you will use the command as below and on the right get an acknowledgement



curl -L https://www.opscode.com/chef/install.sh | sudo bash

now after this we can confirm that the chef has been installed by looking at version by using the below command. 

chef-client -v

and also the folder structure 


Git simplified and master it in 10 minutes!

OK, what is this Git thing! Just because the developers use it, doesn't mean its rocket science.
Some are using it to work on family photo albums so it should be useful for all.

Imagine if you were composing a document and wanted a far off friend's help on it, you would write the initial draft, then email the attachment to your friend, he would download, save a copy, edit it and then email it to you.
Now suppose instead of this emailing overhead, if we could just have this document stored on your computer and your final edits pushed and stored on a web repository which your friend can download from and work on it locally and push the changes to the web repo until its perfect.

Different versions of this document from the initial one to the final one is stored and available and could be called Versions, since the docs are available with everyone working on it and not central its distributed and you and your friend have a set of rules to control the flow, combing it, distributed version control system (DVCS).
The reason for its increased adaptation is 1. No central repo so no point of failure. 2. Open Source

Ok going right into the crux.

Any time you need help on something, and to also check if git it already installed, launch a terminal and type the below, if git is not installed, just google for how to install it on your machine.

  • git help

Just adding our details on Git now so that it know who we are and changes to a doc could be accredited to us


  • git config --global user.name "Venu Murthy"
  • git config --global user.email "xyz@xyz.com" 

Once that is done, lets now go into a directory that you would like to put on Git. mkdir testgit, change to that directory by typing cd testgit


  • git init
Output is "Initialized empty Git repository in /Users/venum/testgit/.git/"

Once you initialised the git, we can check the status of what files have been added or modified by 
  • git status
The workflow is

File created or modified ====staged==> Ready for Save ======commit====> Saved on Repo on disk

So now we need to stage the file i.e. similar to save the modifications to the file on our harddisk. 
  • git add <filename>
filename could be the file name or a wildcard like *.txt or --all to checkin all files in the directory.

once this has been done, we now need to launch the staged file to the repo, its called commit. 
  • git commit -m "some comment--keep it in present tense"
to see the history of what has been done so far
  • git log


To see the changes or differences that have happened after the last commit of a file
  • git diff <filename>
So far we've been able to add files to our repo on disk, now lets push our local "master" to Git source control so that it can be shared. Register on Github.com for eg. Once you create a repo below is how you tell your git on where to push these files
  • git remote add origin https://github.com/venumurthy/testgit.git
  • git push -u origin master
in the above command -u is so that it remembers the parameters so that we wouldn't have to give it the next time. Origin is destination to which master is going to be copied to. 

Go to github and check if your files have come in. After this when you make any changes to the file, remember to do the following:-

  • git add <filename>
  • git commit -m "some comment"
  • git push

Congratulate yourself now for having tamed the octocat! 

OAuth Simplified

In my earlier post we saw what is SAML. Basically it works on the HTTP i.e. for webpages and etc. 
But what if you wanted have SSO on an app, for example on a Mobile App? There are work arounds to implement SAML but the straight forward way of doing it would be to use OAuth, a recent boy in neighbourhood which unlike SAML which is older than 6 years, is designed considering the future, of native apps and mobile apps. 

As usual the definition is 



OAuth is an open standard for authorisation. OAuth provides a method for clients to access server resources on behalf of a resource owner (such as a different client or an end-user). It also provides a process for end-users to authorise third-party access to their server resources without sharing their credentials (typically, a username and password pair), using user-agent redirections.

Please observe the definition, it says Authorisation and not Authentication!


What it means is that if you have an app on your phone and would like it be able to 


OAuth 2.0 is a relatively simple protocol. To begin, you register your application with Google. Then your client application requests an access token from the Google Authorisation Server, extracts a token from the response, and sends the token to the Google API that you want to access.


From Googles doc we see this

OAuth 2.0 is a relatively simple protocol. To begin, you register your application with Google. Then your client application requests an access token from the Google Authorization Server, extracts a token from the response, and sends the token to the Google API that you want to access.



So its basically a user trying to use a web or a standalone mobile or desktop app, which first requests permission from Google servers, and once the user approves this app, the app can fetch resources from the service provider .

One very nice def to lookup is in here.

This is what OAuth does, it allows you the User to grant access to your private resources on one site (which is called the Service Provider), to another site (called Consumer, not to be confused with you, the User). 

At this point in time I would like to give some relief to those who haven't understood it still, relax, seems like Oauth will be obsolete soon.   

SAML (Security Assertion Markup Language) -- Simplified


As discussed earlier SAML provides the web SSO as in the earlier blog on SSO.

By Definition SAML is-

Security Assertion Markup Language is an XML-based open standard data format for exchanging authentication and authorization data between parties, in particular, between an identity provider and a service provider.

Breaking it down.

IdP is the Identity Provider -- someone who has a database containing usernames and passwords. For eg. Google apps.

SP is service provider -- a service we would like to use. For eg. apps like soundcloud and etc which allow us to log into their service using the credentials stored in IdP.

So basically, if you want to add a comment to this blog post, you would have to authenticate with Google apps over the Browser and one Google identifies you as the right user, will allow you to access this resource and leave a comment. This is quite simply SAML at work.

The below video is amazing and a must watch to just understand it further.


For something more detailed see below


SSO--Single Sign On simplified

Before we go there, lets understand what is meant by Authentication and Authorisation.
Lets take an example, if you are watching a video on youtube, its available to public and hence needs not Authentication, but to read your mail on gmail you would need to authenticate, i.e. say who you are and the gmail server checks by comparing your username and password with what has stored, this process is called Authentication. Now suppose you want be able to make changes to a resource, the system checks your Authority to see if are allowed to do that this is authorisation, like the uploader of a video can delete or edit the video where was as we are authorised to only view the video. 

by definition

Single sign-on (SSO) is mechanism whereby a single action of user authentication and authorisation can permit a user to access all computers and systems where he has access permission, without the need to enter multiple passwords. Single sign-on reduces human error, a major component of systems failure and is therefore highly desirable but difficult to implement. 

making it more simple, when you log into Gmail, you can access Google Calendar, Docs and the other 144+ Google services without having to register/signup nor having to log in with username and password, you just go in.... this is SSO or Single Sign On.

SSO can be achieved by SAML or OAuth in the following articles we will see what that means.

Error: SMTPIN_ADDED_BROKEN@mx.google.com

Some times we see the below in the email header information. SMTPIN_ADDED_BROKEN@mx.google.com

The cause of the "...SMTPIN_ADDED_BROKEN@mx.google.com" is because of the 'https://' in the original Message-ID, which is not allowed. There could be some dodo systems which might be doing this and they would need to be tweaked to now allow message-id (see my earlier posts on what it means) and to have https!

Leverage the power of Cloud with Python

My well honoured Presentation at the Cloud Connect International Event in Renaissance Hotel, Mumabi in 2013




Leverage the power of the Cloud using Python from Venu Murthy



It was am amazing moment when my proposal was shortlisted and I was invited to speak in the much famed Cloud Connect event in Mumbai, June 2013.

Below are some pics, the audience were amazing and as one of my friends said, "wow, you are listed with the C level people already, here!"





Google Mail or Gmail threading or conversations demystified



Many of us like the mail threading or in Gamil they are also called as conversations and for those who do not like it, Google does gives us the option to disable threading. To do so, go to the General tab of your Gmail Settings, and select the radio button next to 'Conversation view off'.


 However for those of you who are interested in Gmail threading. Below is a threaded mail




Gmail threading is based on these factors. 

  1. Subject must be same or similar (it can have a prefix of Re:, Fw:)
  2. Sender must be a part of the thread OR in-reply-to must be used. (see this link for the basics) 
  3. Time from when the original was sent and its reply.

A typical header message looks as below:-

Message-ID: <13dsaf693.25715.YahooMailNeo@web126206.mail.ne1.yahoo.com>Date: Mon, 23 Sep 2013 04:44:53 -0700 (PDT)From: Divakar Nadig <xyz@yahoo.com>Reply-To: Divakar Nadig <xyz@yahoo.com>Subject: please see this dear son.....
To: * V e n u * <xyz@gmail.com>

See the below screenshot for a crash course on mails in a thread and their various components.



Advanced version explaining the flow


  • Subject line is the primary factor, which is used to gather candidate threads. The don't have to be exactly be the same, they could have Re:, Fw: and etc but no changes should have been made to the subject of the mail. 
  • These threads are then evaluated to see if the "References" fields (as in the above screenshot) of the header contain message-IDs of threads already in the mailbox. If a reference is found, it is threaded to that reference. 


  • If no references are found, it is then moved to a date-based search, and narrowed down to a list of candidates as follows:
  • The preferred option (typically conversations and automated messages):
  • The incoming message's from address is listed as a to/cc/ from address of a message in the candidate thread, and one of its to/cc addresses is also listed as a to/cc/ from address of a message in the candidate thread. If there are any candidates in the preferred option, it is not even consider the fallback candidates.
The fallback option (typically mailing lists): The from address doesn't match, but one of the incoming message's to/cc addresses is listed as a to/cc/from address of a message in the candidate thread.

If any candidate messages match one of these two options, they are evaluated based on the date of the message. We then choose to thread the message with either the nearest older message or the nearest newer message, whichever is closer, with some important provisos:


There is a different time limit for threading messages that are considered to be a reply or forward (as judged by whether or not the subject has an "re" or "fw" prefix) - these messages can be threaded together with the nearest candidate thread message if it is up to 7 days away.


If the incoming message is not an re/fwd, then we will only thread it with the nearest candidate message if it is within 12 hours away, and also if it is from the same sender.


Whats an email anyway? What are it's components?

A some what weird question but when explored its very interesting and lets see how deep the rabbit hole goes!

Electronic mail is commonly called as e-mail or eMail, its been around from 1973. it’s a method of exchanging digital messages across the internet or other computer networks. After the Snowden era may be we will see a trend of peer to peer communication without a centralised server, the way it was in the beginning of time! 

An Internet email message contains 3 components:-


  • Message envelope
  • Message header – contains control information, flow and etc.
  • Message body
In a client-server model, client is from where the message originate and Email servers accept, forward, deliver and store messages. 

Originally emails could carry text-only (7-bit ASCII) communications medium, which was then extended to carry multi-media content attachments, a process standardised in RFC 2045 through 2049. Collectively, these RFCs have come to be called Multipurpose Internet Mail Extensions (MIME). The Internet email message format is now defined by RFC 5322, got from RFC 2822, 822.

Network-based email exchanged by the SMTP. In the process of transporting email messages between systems, SMTP communicates delivery parameters using a message envelope separate from the message (header and body) itself.

This standard specifies a syntax for text messages that are sent among computer users, within the framework of "electronic mail".

A message consists of header fields and, optionally, a body. The body is simply a sequence of lines containing ASCII characters. It is separated from the headers by a null line.

When we go to an email and analyse its header information, we see the following fields:-

 
TO:This field contains the identity of the primary recipients of the message.  
CC:This field contains the identity of the secondary (informational) recipients of the message.  
BCCThis field contains the identity of additional recipients of the message. The contents of this field are not included in copies of the message sent to the primary and secondary recipients. Gmail choose to include the text of the "Bcc" field only in receipent's copy due to which he got the mail.  
REFERENCE FIELDS 
MESSAGE-ID / RESENT-MESSAGE-IDThis field contains a unique identifier (the local-part address unit) that refers to THIS version of THIS message. The host that generates it guarantees the uniqueness of the message identifier. This identifier is intended to be machine readable and not necessarily meaningful to humans. A message identifier pertains to exactly one instantiation of a particular message; subsequent revisions like replies or forwards to the essage should each receive new message identifiers 
IN-REPLY-TOThe contents of this field identify previous correspondence, which this message answers. Note that if message identifiers are used in this field, they must use the msg-id specification format.A message just sent will not have a IN-REPLY-TO field. 

REFERENCESThe contents of this field identify other correspondence which this message references. Note that if message identifiers are used, they must use the msg-id specification format. i.e. this will have all the Message IDs so far. 
SUBJECTThis is intended to provide a summary, or indicate the nature, of the message. 

See below the Screenshot of the mail threads.



 Some more fields in the mail but not used.
ENCRYPTEDSometimes, data encryption is used to increase the privacy of message contents. If the body of a message has been encrypted, to keep its contents private, the "Encrypted" field can be used to note the fact and to indicate the nature of the encryption. Not applicable when it comes to gmail. 

EXTENSION-FIELDA limited number of common fields have been defined in this document. As network mail requirements dictate, additional fields may be standardized. To provide user-defined fields with a measure of safety, in name selection, such extension-fields will never have names that begin with the string "X-". 

USER-DEFINED-FIELDIndividual users of network mail are free to define and use additional heade fields. Such fields must have names which are not already used in the current specification or in any definitions of extension-fields, and the overall syntax ofthese user-defined-fields must conform to this specification's rules for delimiting and folding fields. Due to the extension-field publishing process, the name of a user- defined-field may be pre-empted Note: The prefatory string "X-" will never be used in the names of Extension-fields. This provides user-defined fields with a protected set of names.

The best IDE for Python -- PyCharm goes opensourced - i.e. Free!!!

It gives me great joy that PyCharm has been opensourced, I've been using it for the past 2 years and am kind of addicted to it, though I do agree that I haven't exploited it enough.

You can download it from this link
Whats so great about it!






Awesome Navigation

Command + Click on a module to go to where the class is defined.
Option + F7 gives a view of where all its used.
Command + Option + B ==> to see where all its implemented.

What I love the most is the way it generates functions for you.

Type name of a method which has not yet been defined. and then
Option + Enter and select Create Function

Option + Command + V will instantiate an object

Enjoy Python!


Simply put whats an API anyway?

API in robotic terms is  application programming interface which ofcourse doesn't make any sense. 

So to understand that in a simple way lets see what is a GUI, again in robotic terms its Graphic User Interface. i.e what ever action you want to take a button or something of the sort could be clicked. Like to close a browser you would take the mouse pointer to the x on the window. 

However if it was the end of your day and you were closing 20 such windows of various apps on your laptop, you know the pain of it. However if you could just automate it! 

Now imagine if there were 2000 such windows to be closed, you would want your computer do something about it but how will you tell your computer on what to do? This is were the APIs come into picture, they are a programatic way of doing manual things on computers. 

Basically APIs are a replacement for GUI if you want to automate a lot of stuff. 

Eg. 

If you are Google apps admin with no LDAP sync and etc and need to create users manually, on a day to day basis, you might be doing the below:-

First open a browser and go to your link, authenticate, go to admin console, then click on Add more users button and 
Select to enter user manually and then get the the fields to enter new user's name.


phew! all this to just reach the screen and what would be the plight if we had to enter 50 users on a given day!
APIs to our rescue!
by simply doing the prilimanaries you can put the below command and have the users created.

gam create user crashtestdummy firstname Crash lastname "Dummy" password "BuckleUp"!


I am late and where was I


I wonder where I was all these days and why did I not blog. And as its human to give excuses, I would say that the companies I worked for never gave me the leisure to either read a blog or post one in peace. But I am going to force myself to share all that I've learnt so far, as this is the best way for me to give back to the internet in my humble what it has given me for more than a decade.