Vert Studios - Programming Blog

[Screencast] C: malloc and functions returning pointers

Wed, 25 Jul 2012 07:45:00 +0000

[Screencast] C: malloc and functions returning pointers

(The video above is optional, and covers no more material than what is covered here. If you like to read and work through things at your own pace, the full article text is below. If you prefer to observe and listen, feel free to enjoy the screencast.)

Before you proceed with this tutorial, you should have a good understanding of pointers. Check out my introduction to pointers screencast.

The ability to manage your own memory is part of what makes C so significant. At the heart of memory management sits malloc, a function which you will know how to use by the end of this tutorial.

As described by the GNU, the malloc function is used to manually allocate a block of memory. Specifically, malloc returns a pointer to a newly allocated block of memory. This brings up the question: What does it mean to return a pointer?

Functions returning Pointers

Similar to how a C function can return an int or char, functions can also return pointers. To demonstrate, let's use the most basic functions possible. First, an int function that just returns the integer passed to it.

// Demonstrate a simple function that returns an integer.
#include<stdio.h>

int return_me(int);

int main(){
  int x = 5;
  printf("x: %d\n", x);

  x = return_me(x);
  printf("x: %d\n", x);

  return 0;
}

/*
 * A very simple function: Just returns the integer passed to it!
 */
int return_me(int num){
  return num;
}

After compiling and running, the output is just

x: 5
x: 5

Now let's take this same idea, but now we'll apply it to pointers.

// Demonstrate a simple function that returns a pointer to int
#include<stdio.h>

int* return_me(int*);

int main(){
  int x = 5;

  // p holds the memory address of the integer x.
  int *p = &x;
  printf("p: %p\n", p);

  p = return_me(p);
  printf("p: %p\n", p);

  return 0;
}

/*
 * A very simple function: Just returns the pointer passed to it!
 */
int* return_me(int *pointer){
  return pointer;
}

The output will be similar to

p: 0x7fff678697fc
p: 0x7fff678697fc

Note the function header int* return_me(int*). We know the structure of a function is

returnType functionName(arg1, arg2, ...){
  // Do stuff
  // Return something of type returnType (unless void!)
}

So it makes sense that if we wanted to return the memory address of an integer, also known as a "pointer to int", we would specify our return type as int*.

For another example, say we have two floating point variables x and y. We want to create a function that returns the address of the larger variable. Here is how we would accomplish this:

// This program contains a simple function that returns a pointer to float.
#include<stdio.h>

float* return_biggest(float*, float*);

int main(){
  float x = 10.0;
  float y = 20.0;

  // Get the address of the variable with the largest value.
  float *p = return_biggest(&x, &y);

  printf("(x) addr: %p, val: %.2f \n", &x, x);
  printf("(y) addr: %p, val: %.2f \n", &y, y);
  printf("The address of the biggest: %p\n", p);
  printf("The value of the biggest: %.2f\n", *p);
  return 0;
}

/*
 * Returns the address of the float variable with the largest value.
 */
float* return_biggest(float *p1, float *p2){
  // biggest is a pointer to float: it will hold the memory address of a
  // floating point variable.
  float *biggest;
  if (*p1 > *p2){
    // biggest is assigned the memory address of the float associated with 
    // the address p1
    biggest = p1;
  }
  else{
    // biggest is assigned the memory address of the float associated with 
    // the address p2
    biggest = p2;
  }

  // return the memory address of the larger float variable
  return biggest;
}

The output will look similar to

(x) addr: 0x7fff57c94728, val: 10.00
(y) addr: 0x7fff57c9472c, val: 20.00
The address of the biggest: 0x7fff57c9472c
The value of the biggest: 20.00

Specifically, the address of y and the address of the biggest should be identical.

Void pointers

We know a pointer to int must hold the address of an int, a pointer to float must hold the address of a float, and so on. However, there does a exist a special pointer type that can be assigned the memory address of any type: These pointers are called void pointers. We declare and assign void pointers just like any other pointer. Observe in the following example how the void pointer can be assigned the address of both a float and an int:

// Void pointers
#include<stdio.h>

int main(){
  int x = 10;
  float y = 20.0;

  void *p = &x;
  printf("p: %p\n", p);

  p = &y;
  printf("p: %p\n", p);
  return 0;
}

With the output looking similar to

p: 0x7fffd3ea8dd8
p: 0x7fffd3ea8ddc

Furthermore, void pointers are assignable to any other pointer type. In other words, a void pointer can be assigned to any pointer variable of any type. To demonstrate:

// Void pointers are assignable to any pointer type
#include<stdio.h>

int main(){
  // Initialize x and y
  int x = 10;
  float y = 20.0;

  // Declare a pointer to int, float, and a void pointer
  int *pint;
  float *pfloat;
  void *vp;

  // Assign vp the memory address of x, a pointer to int.
  vp = &x;
  pint = vp;

  // Assign vp the memory address of y, a pointer to float.
  pfloat = vp;

  return 0;
}

Functions returning void pointers.

If we can declare a variable of type void*, then we certainly can create a function returning a value of type void*, which is simply a pointer of any type.

Consider a previous example which simply returns the pointer to int passed to it. We can modify the function to return a void pointer, and the result will be the same.

// Demonstrate a simple function that returns a void pointer
#include<stdio.h>

void* return_me(int*);

int main(){
  int x = 5;

  // p holds the memory address of the integer x.
  int *p = &x;
  printf("p: %p\n", p);

  p = return_me(p);
  printf("p: %p\n", p);

  return 0;
}

/*
 * A very simple function: Just returns the pointer passed to it!
 */
void* return_me(int *pointer){
  return pointer;
}

It's always better to be explicit when possible, though. Usually you will know exactly what type of pointer you want returned, so for the sake of your sanity (and the sanity of those around you), don't make all your functions returning a pointer be of type void* just to save yourself a few keystrokes.

At this point, void pointers appear to function just like, let's say, a pointer to int. However, there are strict rules and pitfalls that come with void pointers that do not with other pointers.

Dereferencing Void Pointers - It can't be done!

Although void pointers and other pointers have many things in common, the ability to dereference is not one of them. This is because other pointer types tell the compiler how much memory should be read/written when we deference them, but void pointers do not us with this information.

The following program will fail to compile:

// Attempt to dereference a void pointer: This will fail to compile!
#include<stdio.h>

int main(){
  // Get the address of an integer
  int x = 10;
  void* vp = &x;
  *vp = 15;

  printf("The value at the address of vp is: %d\n", *vp);
  return 0;
}

My gcc compiler outputs

ex8.c: In function ‘main’:
ex8.c:10: warning: dereferencing ‘void *’ pointer
ex8.c:10: error: invalid use of void expression
ex8.c:12: warning: dereferencing ‘void *’ pointer
ex8.c:12: error: invalid use of void expression

We can, however, cast a void pointer to the proper pointer type, and then dereference.

// Attempt to dereference a void pointer: Casting works!
#include<stdio.h>

int main(){
  // Get the address of an integer
  int x = 10;
  void* vp = &x;

  printf("The value at the address of vp is: %d\n", *(int*)vp);
  return 0;
}

With the expected output of

The value at the address of vp is: 10

Implicit conversion of void pointers

Luckily, even though we can't dereference a void pointer, we don't have to cast a void pointer if it's being assigned to another pointer. Look at the following example, and reread the previous sentence a few times until it sinks in.

// Demonstrate implicit conversion of void pointers
#include<stdio.h>

int main(){
  // Initialize an integer x
  int x = 10;

  // Assign the address of x to a void pointer
  void *vp = &x;

  // Initialize a pointer to int to vp, which itself is the address of x.
  int *pint = vp;

  // We're free to dereference pint: Even though vp is a void pointer, the
  // assignment above initiated an implicit conversion of vp to a pointer to
  // int, similar to how `float x = 10` implicitly converts 10 to its floating
  // point form. Dereferencing vp at this point would still result in an error:
  // it's never okay to dereference a void pointer.
  printf("*pint: %d\n", *pint);

  return 0;
}

The output:

*pint: 10

We take advantage of the implicit conversion behavior of assigning void pointers to "normal" pointers every time we use malloc, which you'll see in a few minutes.

Playing with Fire

We knew beforehand we can't assign the address of a float to a pointer to int. But now, we know void pointers are implicitly converted to the type of pointer it's being assigned to. If we're feeling smug, we might think:

What if I assign a void pointer holding a pointer to float to a pointer to int?

Indeed, this trickery will compile!

// Void pointers are assignable to any pointers
#include<stdio.h>

int main(){
    // Initialize an integer
  int x = 10;

  // Declare pointers to int and float
  int *pint;
  float *pfloat;

  // Initialize a void pointer to the memory address of the integer x
  void* vp = &x;

  // Since void pointers are assignable to any pointer, the following are both
  // legal (but legal doesn't always mean smart or safe!).
  pint = vp;
  pfloat = vp;

  // This makes sense because pint is a pointer to int, and vp holds the
  // memory address of an integer.
  printf("*pint: %d\n", *pint);

  // But this doesn't make sense because pfloat is a pointer to float, but
  // vp holds the memory address of an integer.
  *pfloat = 10.0;
  printf("*pfloat: %f\n", *pfloat);

  // Now our data has been corrupted. Lesson learned: Just because void
  // pointers are legally assignable to everything doesn't mean you're safe!
  printf("*pint: %d\n", *pint);

  return 0;
}

Output will appear similar to

pint: 10
pfloat: 10.000000
*pint: 1092616192

This behavior is explained in this StackOverflow thread,

... dereferencing a pointer that aliases another of an incompatible type is undefined behavior. Unfortunately you can still code this way, maybe* get some warnings, have it compile fine, only to have weird unexpected behavior when you run the code.

One thing that you will learn about C, if you haven't already, is that legal does not necessarily imply correct, and correct now does not necessarily imply correct forever! Be careful.

Allocating Memory with malloc

If you look closely at the previous examples when functions returned pointers, we've only returned pointers that were declared in main() and passed to the function. What if I wanted a function to return the address of a variable declared inside the function?

What not to do

We know that initialization statements like int x = 10 allocate memory, and that allocated memory has an address. Consequently, the following example appears to be a sufficient way to allot memory for a variable and return its address.

// Example of how NOT to allocate memory!!!
#include<stdio.h>

int* new_integer(void);

int main(){
  // Get the address of an integer
  int *p;
  p = new_integer();

  printf("The value at the address of p is: %d\n", *p);
  return 0;
}

int* new_integer(void){
  int x = 10;
  return &x;
}

Returning the address of a local variable, however, is undefined behavior. It might work as "expected" for a short period of time and fail later. Why is that? This comment from /r/programming explains the issue very well:

Once the function [a local variable] is defined in returns, the memory used for storing that variable will be "freed" and almost certainly reused at some point in the future. Returning a pointer to a [local] variable ... means that you are returning a pointer to memory that can be overwritten at any time. Reading or writing to that variable can cause any number of bad things to happen, including data corruption and crashes. Worse, it may work fine, at least for a while, making it very difficult to debug.

It is mandatory that you understand the heap and the stack, but for this tutorial, the explanation above will suffice.

What to do - malloc!

Now that we understand how not to allocate memory, we can discuss how you should allocate memory. For this tutorial, we will use the malloc function for allocating memory, although other allocation functions exist.

The definition of malloc is as follows:

void* malloc (size_t size)

This function returns a pointer to a newly allocated block size bytes long, or a null pointer if the block could not be allocated.

For all intents and purposes, you can consider size_t (read "size type") as an unsigned integer. (More discussion on size_t vs int)

So when calling the malloc function, you specify how many bytes of memory you want allocated, and if malloc is able to allocate that memory, it returns a void pointer to that memory block.

Let's take our incorrect example and fix it by implementing malloc.

// Allocating memory with malloc
#include<stdio.h>
#include<stdlib.h> // required to use malloc

int* new_integer(void);

int main(){
  // Get the address of an integer
  int *p;
  p = new_integer();
  *p = 15;

  printf("The value at the address of p is: %d\n", *p);
  return 0;
}

int* new_integer(void){
  // Allocate the exact amount of memory needed for an integer via sizeof(int).
  // malloc returns a void pointer, but the assignment to `int *pointer` causes
  // an implicit conversion to type int*.
  int *pointer = malloc(sizeof(int));
  return pointer;
}

Be sure to note the inclusion of the stdlib.h library, as well as the usage of the sizeof operator. sizeof tells us the number of bytes a datatype or variable takes up in memory.

While this example demonstrates how to use malloc, it's not particularly useful in any real world scenario. malloc really begins to shine when we start creating data structures such as linked lists, stacks, queues, binary trees, and more. The process of creating linked lists, which will be the subject of my next article, frequently involves allocating memory for a structure. Here's an example of dynamically allocating memory for a structure using malloc:

// Using malloc to allocate memory for a structure
#include<stdio.h>
#include<stdlib.h>

// Create a 'rectangle' structure.
typedef struct{
  int height;
  int width;
} rec_t;

rec_t* new_rectangle(void);

int main(){
  // Declare a pointer to a rectangle structure. `ptr` will hold the memory
  // address of a structure.
  rec_t *ptr;
  ptr = new_rectangle();

  // Now that ptr has the memory address of a rectangle structure, we can
  // perform operations on *ptr just like we would any other structure.
  // In this case, assignment to another structure variable.
  rec_t rectangle = *ptr;

  rectangle.width = 10;
  rectangle.height = 10;

  // Display the dimensions.
  printf("The height: %d\nThe width: %d\n", rectangle.width, rectangle.height);

  return 0;
}

rec_t* new_rectangle(void){
  // Allocate the exact right amount of memory for a rectangle structure.
  // Due to how the sizeof operator works, we can use a "shortcut" and get
  // the number of bytes needed without having to specify the rec_t type to
  // sizeof. 
  rec_t *p = malloc(sizeof *p);
  return p;
}

Take a good look at the new_rectangle function, and take a couple minutes to just memorize that initialization of p. It is not necessary to specify rec_t, because the compiler knows that *p is of that type. It might seem strange that we can refer to *p in the initialization of p, but we can! This type of call to malloc is much more concise and accurate.

In the proceeding example, where we used malloc(sizeof(int)), we could have used malloc(sizeof *pointer) instead.

In many tutorials/discussion forums, you probably would have seen the new_rectangle function in the following form:

rec_t* new_rectangle(void){
  rec_t *p = (rec_t*)malloc(sizeof(rec_t));
  return p;
}

Earlier I claimed that "explicit is always better". The above, however, is overly-verbose to the point of being harmful. Keep it simple!

Freeing Allocated Memory

When you no longer need memory you've allocated with malloc, you can pass the address returned by malloc to the free function, and that memory will be freed up.

// Allocate memory, store the memory block address in rec
rec_t *rec = malloc(sizeof *rec);

// Do stuff with rec
// ...
// ...
// ...

// Free the memory block located at the memory address `rec`
free(rec);

What comes next?

With a basic understanding of functions that return pointers and how to use malloc, and a little more knowledge of the interactions between pointers and structures, we'll be on our way to creating data structures!

[Screencast] Beginner's Introduction to Pointers in C

Wed, 18 Jul 2012 11:03:00 +0000

[Screencast] Beginner's Introduction to Pointers in C

Pointers aren't that hard. You might find that hard to believe, but it's true! Unfortunately, the simplicity of pointers tends to get lost in poor or overly-verbose explanations. This is my attempt to present pointers in a clear, easy, and accessible manner.

This screencast answers the following question:

What is a pointer, and why are they important?
How do I declare a pointer?
What does it mean to 'dereference' a pointer?
How do I handle pointers inside a function?

Lock TwitchTV in Landscape Mode on your iPhone

Sun, 15 Jul 2012 03:27:00 +0000

Lock TwitchTV in Landscape Mode on your iPhone

One of the most annoying aspects of iOS is the lack of a straightforward way to lock your iPhone in Landscape mode. Portrait mode lock works just fine, but if you're like me and enjoy watching TwitchTV in bed, Portrait mode isn't that great for watching your favorite streams. If you attempt to watch streams without a lock of some form, though, you'll find your screen constantly switching orientations.

However, landscape lock does exist, it just takes a bit of work to activate!

Step by Step Guide

Open the Settings application. Scroll down and Press General. Next, scroll down and press Accessibility.

Scroll down again until you see the AssitiveTouch and Triple-click Home options. Set AssitiveTouch to ON and Triple-click Home to AssitiveTouch.

The AssitiveTouch icon should pop up on your screen. If you don't see it, press the Home button on the bottom of your iPhone three times in quick succession. After about five seconds, the icon should appear. Pressing the Home button three times like this toggles the AssitiveTouch icon.

Next, lock your iPhone orientation in Portrait mode.

Now open up the TwitchTV app, and pull up your favorite stream. Press the AssitiveTouch icon. A menu will pop up with the labels "Gestures", "Favorites", "Device", and "Home". Press Device.

Another menu will come up. Choose Rotate Screen.

Finally, choose either Right or Left (Left to have the iPhone volume buttons on top).

Now TwitchTV is locked in Landscape mode! Press anywhere outside of the AssitiveTouch menu to minimize it, and feel free to toggle AssitiveTouch off until you need it again.

A few things to remember

You will need to repeat this procedure any time you leave the TwitchTV App.
You are able to drag the AssitiveTouch icon around on the screen.
For whatever reason, toggling AssitiveTouch can take a really long time. If AssitiveTouch fails to show up after triple-pressing the Home key, wait 10 seconds before trying again.

How to Read a Programming Tutorial

Fri, 13 Jul 2012 01:16:00 +0000

How to Read a Programming Tutorial

Many of us seek to improve ourselves beyond what's required. Many CS students are not content to just make good grades in class; they want to get their hands dirty with real-world projects. Many employees read up on the latest languages and frameworks well beyond office hours, simply because they want to play with technology while learning to build a better product. It's important, however, to ensure our time spent studying new programming techniques does not go to waste.

!(confused) != knowledgeable

Understanding an article does not necessarily grant you the knowledge to apply it directly to your projects. I feel it's easy to mistake lack of confusion for thorough understanding. In other words, you can glaze over an article and not feel particularly puzzled by what you just read, yet still have no idea how to reproduce any examples the moment you close the browser window. I use the following techniques to help me determine if I've truly understood what I've read.

Ask Questions. Test Yourself!

if you come across something that doesn't sit right in your head, don't just think "Hmm, okay, I guess that makes sense." Try to figure out why, things work, even if that involves moving beyond the article and searching through Google. Attempt to make slight adjustments to provided examples. Change the problem requirements if you're reading a book with provided exercises. The goal is deep, applicable understanding of the subject, and changing examples and problems will expose you to more edge cases, exceptions, and greatly accelerate your understanding. Here are some examples:

A C tutorial teaches you how to sort an array with hard-coded elements. Can you alter the example to read inputs from the keyboard? How would you account for fewer/more inputs than the array size when sorting?
A git tutorial teaches you how to revert a commit. How could you revert multiple commits at the same time if needed?
A rails tutorial has you storing user passwords using a SHA variant. Can you alter the tutorial to implement bcrypt?

Complete the examples, and then attempt to differentiate minor details from the core concepts. Research the core concepts, and then apply the concepts by changing up the minor details.

Could you recreate what you just read?

I touched on this earlier, but I firmly believe one should have the ability to recreate examples from tutorials from memory. I would even go so far to claim entire projects/applications, such as the Twitter clone from Hartl's free Rails book, should be recreated from memory. Understanding code after glancing through it does not in any way guarantee you can replicate it, no matter how "trivial" you think it is. Replicating code from memory forces you to think of the problem requirements, the caveats, and it effectively exposes the concepts you didn't even know you were unsure of.

Be Thorough, Be Honest

Reading programming tutorials effectively can be summarized in two words: thoroughness and honesty. Ask questions, alter examples, and never let your ego pressure you to avoid topics you don't understand.

Resize a Visual selection block to 80 width in VIM

Fri, 13 Jul 2012 12:11:00 +0000

Resize a Visual selection block to 80 width in VIM

Quick VIM tip today: Resizing a Visual selection block.

Suppose you have a paragraph of text in a markdown document. You want this paragraph to be composed of lines with a maximum width of 80.

First, set your textwidth to 80 via :set tw=80.

Now you can quickly resize the paragraph by selecting it in Visual mode and executing gq.

Watch the demo below!

Structures in C Tutorial: The Basics

Tue, 10 Jul 2012 11:36:00 +0000

Structures in C Tutorial: The Basics

This post serves as an introduction to using Structures in C! This is the first of many C tutorials I plan on writing.

Reading this Tutorial

This tutorial is not meant to be quickly glossed over, unless of course you are already familiar with the subject matter. Each full program (i.e. any snippets beginning with #include) should be hand-typed, compiled, and ran. I also encourage you to try to type the example programs entirely from memory once you understand them; doing so helps me store new-found knowledge into long-term memory, and perhaps you'll receive the same benefits!

What is a Structure?

As said by Neol Kalicharan in Data Structures in C:

"In C, a structure is a collection of one or more variables, possibly of different types, grouped together under a single name for convenient handling."

Structures are pretty simple to understand just by seeing an example in action. Compile and run the following:

// Using structures in C
#include<stdio.h>

// Struct declaration
struct person{
  char *name;
  int age;
  char gender;
};

void print_person(struct person);

int main(){

  // Declaring a variable of type `struct person`
  struct person someguy;
  someguy.name = "Joseph";
  someguy.age = 21;
  someguy.gender = 'M';

  // Print person information
  print_person(someguy);

  return 0;
}

// Print a person's information
void print_person(struct person p){
  printf("Name: %s\n", p.name);
  printf("Age: %d\n", p.age);
  printf("Gender: %c\n", p.gender);
}

Explanation

In the Struct declaration section, we define a person type as a structure. We then define the fields of the structure:

a pointer to char (to hold a string literal)
age as an integer
gender as a char.

Fields of the structure are accessed via the dot operator.

#include

struct product{
  char *label;
  int value;
};

int main(){

  struct product cookie;
  cookie.label = "wonderful cookie";
  cookie.value = 1;

  printf("The %s is $%d\n", cookie.label, cookie.value);

  return 0;
}

After declaring the person structure, we can now do the same things with struct person as we would regular data types, such as int.

// Declaring a variable
struct person someguy;
int x;

// Usage in function parameters
void print_person(struct person p){
  ... // Do things with p
}
void print_number(int num){
  ... // Do things with int
}

// Assignment
struct person someotherguy = someguy;
int y = x;

// Operations
int ageDiff = someotherguy.age - someguy.age;
int z = x - y;

Tidying up with typedef

If you'd like to eliminate the verbosity of using struct person every time you're working with a person structure, you can use typedef.

typedef works like so:

typedef datatype newDataTypeName

For example,

// Using typedef
#include<stdio.h>

typedef int WholeNumber;

int main(){
  WholeNumber x = 10;
  printf("%d is a whole number\n", x);
  return 0;
}

While this example isn't particularly useful, it lets you understand how typedef works.

Note again that struct person behaves like any other data type, so the entire structure declaration can be used as the datatype parameter of typedef. This really cleans up structure usage.

// Using typedef
#include<stdio.h>

typedef struct product{
  char *label;
  int value;
} product_t;

int main(){
  product_t cookie; // Was formerly struct product cookie
  cookie.label = "wonderful cookie";
  cookie.value = 1;

  printf("The %s is $%d\n", cookie.label, cookie.value);
  return 0;
}

Looking again at the usage of typedef (typedef datatype newDataTypeName), we see that struct product{..} is the datatype, and Product is the newDataTypeName. We can now use product_t in place of struct product. This includes variable assignments, function parameters, everything. (Note that product_t is read "product type", and structures generally follow this naming convention.)

Applying typedef

With our newfound knowledge of typedef, lets implement it into our first program. (This is definitely a code snippet I would attempt to type from memory)

// Using structures in C
#include<stdio.h>

// Struct declaration
typedef struct person{
  char *name;
  int age;
  char gender;
} person_t;

void print_person(person_t);

int main(){

  // Declaring a variable of type `person_t`
  person_t someguy;
  someguy.name = "Joseph";
  someguy.age = 21;
  someguy.gender = 'M';

  // Print person information
  print_person(someguy);

  return 0;
}

// Print a person's information
void print_person(person_t p){
  printf("Name: %s\n", p.name);
  printf("Age: %d\n", p.age);
  printf("Gender: %c\n", p.gender);
}

Conclusion

Structures are great for representing collections of related attributes, and they take very little effort to use! We'll cover more involved uses of structures, such as arrays of structures and pointers to structures, in an upcoming post.

Converting a simple website to Python Flask

Mon, 09 Jul 2012 10:17:00 +0000

Converting a simple website to Python Flask

Flask is a Python web framework that's steadily increasing in popularity within the webdev world. After reading so many great things about Flask, I decided to try it out myself. I personally find testing out a new framework difficult, because you must find a project complex enough to reveal the framework's quirks, but not so daunting as to take the fun out of the project. Luckily, my PHP/Wordpress powered website filled this role quite nicely - the website simply consists of static content, a contact page, and a blog. If I could not convert such a simple site over to Flask, I would immediately know that Flask and I would not make a good team.

Spoiler alert: You're reading this on a Flask powered site! Feel free to check out the source of this site on Github.

Getting Started

The first task at hand was simply getting the home, about, services, and work pages to render correctly. While the task consisted mostly of copying and pasting, I was able to immediately apply most of the concepts from the Flask quickstart guide. Specifically, I had to learn

How do I actually start the server?
Where do my templates go?
Where do my static assets go?
How do I retrieve my assets from within the templates?
How do I route requests?

Sure enough, getting the pure HTML documents to render was pretty simple. By pure HTML, I mean there were no includes for headers, footers, etc. Thus the logical next step was to take these pure HTML documents and DRY up the repeating areas.

Using Jinja2

UPDATE: Turns out I was just being dumb. The include statements work fine.

Jinja2, Flask's default templating engine, uses template inheritance in additon to supporting includes. To demonstrate, suppose we have a parent template layout.html

<!DOCTYPE html>
<html lang="en">
  <head><title>My Website</title> </head>
  <body>
  <div id="wrapper">
    <div id="content">
      {% block body %}{% endblock body %}
    </div><!--content-->
  </div><!--wrapper-->
  </body>
</html>

And a child template index.html that inherits the parent and fills in its blocks

{% extends "layout.html" %}
{% block body %}
  Body content goes here!
{% endblock body %}

If we were to call render_template on index.html, this is what we would get.

<!DOCTYPE html>
<html lang="en">
  <head><title>My Website</title> </head>
  <body>
  <div id="wrapper">
    <div id="content">
      Body content goes here!
    </div><!--content-->
  </div><!--wrapper-->
  </body>
</html>

The include statement works similarly to PHP's include statement in regards to importing files that render markup. Consider the footer for this site, located in templates/includes/footer.html:

<div id="footer">
  <div class="hr"><hr /></div>
  <p>
    <a href="mailto:hi@vertstudios.com">hi@vertstudios.com</a> |
    <a href="http://www.twitter.com/vertstudios">@vertstudios</a>
  </p>
</div>

And templates/layout.html:

<!DOCTYPE html>
<html lang="en">
  {% include "includes/meta_and_css.html" %}
  <body onload="prettyPrint()" id="{{g.bodyID}}">
<div id="wrapper">
  {% include "includes/nav.html" %}
  <div id="content">
    {% block body %}{% endblock body %}
  </div><!--content-->
  {% include "includes/footer.html" %}
</div><!--wrapper-->
{% include "includes/javascripts.html" %}
</body>
</html>

Being 'Brave'

Flask didn't always have an out-of-the-box solution to the problem at hand. In fact, it rarely did. Flask did, however, give me all the tools necessary to create whatever I wanted. In a way, I felt that Flask itself was telling me "Joseph, you're smart enough to implement X functionality on your own... be brave for once!"

Sometimes reinventing the wheel is a rewarding exercise.

The Contact Form

The recommended method of dealing with forms in Flask is the Flask-WTF extension. Flask-WTF has many built in validation methods, but it did not include validation for a phone number. No big deal, any form library worth its salt has means for defining custom validation functions. WTForms does in fact have a mechanism for implementing custom validators, but I did not find their examples very pleasing to the eye:

def length(min=-1, max=-1):
    message = 'Must be between %d and %d characters long.' % (min, max)

    def _length(form, field):
        l = field.data and len(field.data) or 0
        if l < min or max != -1 and l > max:
            raise ValidationError(message)

    return _length

class MyForm(Form):
    name = TextField('Name', [Required(), length(max=50)])

Consequently, I used Flask-WTF to gather data from the form fields, but I rolled my own validation module, which only took about an hour.

The actual sending of the email was easily handled with Python's built-in smtplib module and Gmail.

The Blog

Creating a blog has become the "Hello, World" of web application frameworks. While Python blogging software does exist, it seemed like it would take longer to integrate an existing blogging platform into my Flask project than just rolling my own.

For the past year or so, I've longed for a file-system based blogging platform. Writing my articles in VIM + Markdown is so much more enjoyable than writing HTML in VIM, moving the HTML over to Wordpress, previewing the HTML, making sure the formatting is right, and other annoying steps.

There were a few specific things I needed to accomplish with the blog:

Ability to generate an RSS feed
The URLs of the blog posts must be the same as before.
Pagination

Without the need to generate an RSS feed, the only thing I would need to do is route /blog/<post>/ to a function that attempts to render a template named /blog/the-post-url, and return a 404 if that template isn't found. The RSS requirement, however, forced me to think how I should go about storing the meta data for each post.

I settled on a fairly simple structure that, while not beautiful or clever, works good enough for me. I create a directory in /blog/posts/ named desired -post-url. Within that directory, I create two files: meta.py and body.html. meta.py just contains a few variables for metadata (date posted, excerpt, etc), and body.html has the post body.

To avoid having to parse through all the metadata files to figure out the order of the posts for the RSS feed, I just prepend the directory name to the top of a file called rss.txt. Again, not the most beautiful thing in the world but it's a system that I find easy to understand and easy to work with.

The RSS feed and blog pagination are handled in the same step. I run a script called genfeed.py which creates the rss xml file and pagination files. Using static files generated by the geenfeed.py script (which reads the rss.txt script mentioned earlier) for pagination makes sense because the pagination should only change whenever I update/create a story.

Importing my Blog Posts

To import my Wordpress posts into this new structure, all I had to do was parse Wordpress's export xml file with PyQuery and create the files with standard library functions. I didn't bother to convert the posts from Wordpress to Markdown, so all old posts you look through in the source will have normal HTML.

Structural Troubles

One of the biggest troubles I faced throughout the site development was how to import things into particular files. This StackOverflow answer describes the steps necessary to import a module in a parent/sibling directory when running a script directly. Consequently, any scripts that needed access to the app object (which holds a lot of important methods and attributes about the application instance), I had to either run python -m mywebsite.blog.genfeed, or just move genfeed.py to the same directory as mywebsite.py, which would enable me to run python genfeed.py. I chose the latter out of sheer laziness, though I'm not particularly happy I had to make such a choice. I should note that this is not a Flask specific issue, but a Python issue.

Development vs Production Environment

In development, all assets are served from the static files. In production, all assets are served from S3. The environment is set by a env-var called FLASK_ENV. I needed a way to let my templates know "Hey, you're in the production environment, you should link to S3 files instead of static files!". I decided to take advantage of Flask's g object, which is a global object that exists througout the request context. You're free to populate it with whatever you want. I created and used the property g.assets.

While this suited my needs perfectly, it caused my templates to appear slightly less pretty. Anytime I wanted to call a macro that looks for the g.assets property, I'd have to pass g to it. For example, here's the definition of the img macro, a function used to render images on the site:

{% macro img(g, file, alt="", class="") %}
<img 
  src="{{g.assets}}/images/{{file}}" 
  alt="{{alt}}" 
  class="{{class}}" 
  />
{% endmacro %}

An example call to the macro:

{{img(g, "pointer.png", "", "pointer")|safe }}

It's more verbose than what I'd really like, but it gets the job done.

Going Live with Nginx and uWSGI

Now that my project was pretty much finished, it was time to put the files on my Linode VPS and show my project to the world! I decided to use Nginx with uWSGI, simply because the pair have been the subject of many tutorials. Flask itself has documentation on uWSGI. However, I found parts of the documentation inadequate.

From the documentation:

# Given a flask application in myapp.py, use the following command:
$ uwsgi -s /tmp/uwsgi.sock --module myapp --callable app

If you want to put a project on a live server, there's no way you should have to do so much work! uWSGI has ini configuration files that I took full advantage of. Now, instead of executing a verbose command from the shell, I can simply execute

$ uwsgi uwsgi.ini

My uwsgi.ini is as follows:

[uwsgi]
project = vertstudios
daemonize = /var/log/nginx/access.log
master = true
chdir = /var/www/vertstudios.com
socket = 127.0.0.1:5000
wsgi = %(project):variable holding app instance
virtualenv = env/
pidfile = /var/run/uwsgi/%(project).pid 
touch-reload  = /var/run/uwsgi/%(project).pid  
processes = 3 
procname-prefix = %(project)

Now my uWSGI configuration is under version control with the rest of my project.

Here's an explanation for the parameters:

project: The module name where you call app.run().
daemonize: If you want to daemonize. For some reason, you have to specify a log file path or uWSGI will just shove things down stdin. true did not work for me.
master: Boolean indicating if you want a master process. Master processes make uwsgi management much easier.
chdir: Change directory. Essentially lets you keep parameters such as "project" and virtualenv relative to the path provided to chdir.
socket: A Unix socket path or TCP socket info
wsgi: %(the module containing app):app
virtualenv: Path to the virtualenv for the project.
pidfile: Where the master process pid file will be written.
touch-reload: Touching this file causes the processes to reload (convenient!)
processes: How many processes you want running (not including master process)
procname-prefix: Process name prefix. Useful for ps aux | grep myapp

Now, for the parts of my nginx config relevant to uWSGI:

# Redirect everything to the main site.
server {
  server_name www.vertstudios.com;
  return 301 http://vertstudios.com$request_uri;
}

server {
  server_name vertstudios.com;
  location / { try_files $uri @flaskapp; }
  location @flaskapp {
    include uwsgi_params;
    uwsgi_pass 127.0.0.1:5000;
  }
}

Things left TODO

I felt I had done enough to merit scrapping my PHP/Wordpress site and put the Flask site up in its place. Despite that, there's still quite a bit of work to be done.

I plan on exporting my Wordpress comments to Disqus. I enjoy interacting with people that read my articles, and I'd like to preserve the comments I've already acquired.
I've managed to break a few links. In particular, I've broken links to php files I stupidly used to demo some of my jQuery plugins.
Using prettify for syntax highlighting means I have to still use <pre> tags in my documents, since the script requires a class of "prettyprint" on those tags. I'll either adjust prettyprint js and styles myself, find an alternative syntax highlighter, or do some processing of the markup to append the "prettyprint" class whenever it finds the <pre> tag.
Still need to set up git deployment. I'm currently just pulling from a private bitbucket repo on my server.
I need to automate my build workflow. A single bash script should concat all my CSS/JS files, run YUICompressor, and upload the files to S3.
I really need to implement meta description tags for my posts...but that would require me to actually write them.

Overall Impressions

As a whole, I'm quite pleased with Flask. For the most part, it gets out of your way and lets you build things how you want to build them. Consequently, your project structure may be sloppy (especially when first learning Flask), but you're guaranteed to understand how and why everything works.

Markdown to PDF Utility

Thu, 03 May 2012 22:57:00 +0000

Markdown to PDF Utility

I needed to get a Markdown document in PDF format. Unable to find a good solution, I set off to make a Markdown to PDF conversion utility. About 20 lines of Python later, here we are!

md2pdf: Markdown to PDF

You can download md2pdf on Github. Installation instructions can be found in the README. Usage is extremely simple:

$ md2pdf my-markdown-document.md

How it was made

This utility is actually pretty silly. It simply serves as a glue between markdown2, which takes Markdown and produces HTML, and xhtml2pdf, which takes HTML and produces PDFs. The script uses a system call because the xhtml2pdf interface is non-trivial. Consequently, you're probably better off not using this on a production server.

Ubuntu Screencast Software for 11.04+

Wed, 25 Apr 2012 05:07:00 +0000

Ubuntu Screencast Software for 11.04+

I browsed Google for quite a while before I finally found a good product for recording screencasts on Ubuntu. Most software I found, no offense gtk-recordMyDesktop, was extremely outdated and hardly functional. Video quality was either terrible or video/audio was out of sync.

Enter Kazam!

Thanks to some more strict queries on Google, I found Kazam on AskUbuntu.com. It has an extremely simple interface, as seen below. The quality is just fine for simple screencasts, and it allows you to record audio that actually syncs with the video!

Installation

First, we need some Python modules to be installed.

$ sudo apt-get install --reinstall gir1.2-gtk-2.0 python-software-properties software-properties-gtk

Now we actually grab Kazam as instructed by their site.

$ sudo add-apt-repository ppa:kazam-team/stable-series
$ sudo apt-get update
$ sudo apt-get install kazam

And you're done! You can run kazam via the command line or your distro's menu.

Extras

If you happen to have FFMpeg installed, here's a nice 2-pass video/audio encoder script that encodes with H.264.

VI Keybindings in your Shell

Wed, 25 Apr 2012 03:52:00 +0000

VI Keybindings in your Shell

There was once a very sad point in my development career where if I made a typo earlier in one of my bash statements, I would have to hold the left arrow to get my cursor over to the problem area and fix it. Then, one fateful day, I discovered VI keybindings in the shell!

Activating VI Keybindings

To get VI keybindings working in your shell, simply execute

$ set -o vi

(If you decide you like vi keybindings in the shell and you want this mode enabled upon shell startup, append the command to your ~/.bash_profile for OS X and ~/.bashrc for linux ((Why the difference?))

Using the VI Keybindings

The shell will have normal/insert mode as expected. Here are the commands I use frequently:

i - enter insert mode
esc - back to normal mode

* IN NORMAL MODE *
cc - clear the current line and enter insert mode
 C - clear from cursor until the end of the line and enter insert mode
dd - delete the current line
 D - delete from the cursor until the end of the line
 w - next word
 W - next white-space delimited word
 b - previous word
 B - previous white-space delimited word
dw - delete word
cw - change word
fx - find the next occurrence of the character 'x'
Fx - find the previous occurrence of the character 'x'
 $ - move the cursor to the end of the line
 0 - move the cursor to the beginning of the line
 ; - repeat last find command (preserves direction)
 l - move cursor right
 h - move cursor left
 k - scroll up through command history
 j - scroll down through command history
 A - move the cursor to the end of the line and enter insert mode
 I - move the cursor to the beginning of the line and enter insert mode

* Numeric modifier keys work as well! *
5W - move forward five white-space delimited words.