I wanted to read the article "Idempotence Is Not a Medical Condition" by Pat Helland in the May 2012 issue of CACM because it smelled like an article that was full of big words and fuzzy architecture abstractions. I learned some things from the article (and I really like the title), but it's mostly full of FUD. The jist of the article is: "Are you SURE that the messages you send over the network are delivered successfully? Are you REALLY sure? I mean, are you SUPER DUPER sure? After all, how can we be certain of anything in this crazy world?" By the way, SQL Server Broker eliminates this uncertainty for you.

In this article, Helland says that distributed systems today are largely composed of a collection of off-the-shelf software and cloud-based Internet services. This means that they lack a centrally-enforced communication policy. So, extra care must be taken to ensure that each message is delivered and that no message is lost, even if one of the sub-systems goes down or restarts.

This contrasts with the traditional notion of a distributed system: a cluster of tightly-coupled computers in a lab somewhere, which are specifically programmed to talk only to each other. Communication is simpler in this situation because each sub-system is essentially identical and also physically present in the same location.

Every sub-system has some sort of "plumbing" which sends and receives messages (e.g. a TCP network stack). If the plumbing is able to do things like handle duplicate messages and resend lost messages, then the system's application layer does not have to be programmed to ensure reliable communication. But if the plumbing cannot do these things, then the application layer is responsible.

A message should only be marked as "consumed" if the action that the receiving application performs on the message is successful. For example, if an application tries to save a message it receives to a database, but the database transaction fails, then the message should not be considered "consumed" because the operation that the message invoked (a database call) was not successful. The message should sit in a queue somewhere until the application is able to successfully process it or it should respond to the client saying that it failed the operation.

Helland also states that messages should be idempotent, meaning that if the same message is sent multiple times, the effect it has on the server should be the same as if just one message was sent. The term is often used in the context of HTTP--GET requests are defined as being idempotent, while POST requests are not.

Messages can have a preference for one of two types of behavior: "history" or "currency". History means that the messages must be delivered in order. If message N+1 arrives, but message N has not yet arrived, then it must wait until it receives message N in order to deliver message N+1 to the application. Downloading a file requires this type of communication because all of the file's data must be delivered in order or else you'll get a corrupt file. Currency means that getting the most recent message is what matters. If some messages are skipped or lost, that's OK. Getting the most recent price of a stock is one such example of this behavior type.

Helland says that the greatest moment of uncertainty during the request/response cycle is right after the request is sent and the application is waiting for a response. The assumption is that the message has been received and is being processed, but you don't know this for sure. For all you know, the receiving end could have decided to ignore the request and not send a response. It's for this reason that timeouts are needed. If the requester has not received a response within the timeout period, then it will assume the request was not processed and will stop waiting for a response.

However, there is one thing that you can know for certain, Helland says. "Each message is guaranteed to be delivered zero or more times!". Finally, something I can rely on!

Helland says that despite TCP's robustness and wide-spread use, it cannot fully be trusted to deliver messages reliably. He says that TCP "offers no guarantees once the connection is terminated or one of the processes completes or fails." I don't understand what he's trying to say. Of course it offers no guarantees once the connection is terminated. You can't send any messages over a terminated connection. And of course it offers no guarantees once one of the processes completes or fails. At that point, the TCP conversation is over.

He then goes on to say that "challenges arise when longer-lived participants [such as HTTP requests] are involved." He says that when a persistent HTTP connection is needed, the TCP connection is usually kept alive, but there's no guarantee that this will happen. Because of this, the HTTP request may have to be sent multiple times.

Helland then goes into an in-depth discussion about idempotency. He says that, technically, no idempotent request is truly idempotent because every request has some lasting effect on the server. For instance, most servers keep an access log of every request that was received. Making five identical requests, even if they are idempotent, will add five entries to the log file. Also, the performance of the server is impacted every time it receives a request. The more requests it receives, the more its performance will degrade. However, these side-effects are not related to how the actual application logic behaves, which is the true context in which the term "idempotent" should be used.

Helland states that state-ful communication is more difficult than state-less communication because all previously sent messages must be taken into consideration when processing the current message.

He also points out that there's no way of knowing whether the server that receives a request is doing the actual work to fulfill that request. The server could be forwarding the request to another server to do the actual work.

Helland says that if a dialog between two servers breaks apart in the middle of the conversation, both ends must be able to cleanly recover from this failure.

If a service is load-balanced across multiple servers, then state-ful information must be stored in such a way so that a client's state is not lost. One way to do this is to store the state information on a designated server that the other servers have access to. That way, requests from the same client can be handled by any server in the cluster.

Alternatively, the client could be assigned to a designated server in the cluster by a load-balancer when it makes its first request. This server will then be responsible for maintaining the state information for this client and handling all of the client's requests. The first request that the client makes must be idempotent because if the request is received successfully, but the response from the server is lost, then the client will assume that the request was lost and try making the request again. When the request is sent for the second time, the load-balancer may assign the request to a different server. If the request is not idempotent, then the request will be applied twice, thus tainting the server's data. Once the client receives a server response to its first request, it now knows which server in the cluster it should communicate with, which means that all subsequent messages don't have to be idempotent because there's no risk of sending the same request to multiple servers.

Helland says there are three ways to make this first client request idempotent. (1) You can send basically an empty message to the server, such as a TCP SYN message, (2) you can perform a read-only operation, or (3) you have the server queue a non-idempotent operation which will only be executed by the server once the connection has been confirmed. Approach (1) is the simplest, but approaches (2) and (3) can be seen as more efficient because they are performing a useful operation. Or, in Helland's words: "allowing the application to do useful work with the round-trip is cool."

The last point Helland makes is that the last message of a conversation cannot be guaranteed. This is because, if you were to send a response to the last message stating that you have received it, then it wouldn't be the last message! Therefore, applications must be designed so that it is not important whether the last message is received or not.

I recently added database migration functionality to my Sleet SMTP project. This means that, if I release a new version of the application that includes a change to the database schema, the existing databases of deployed applications will be migrated to the new schema automatically. Before, you would have had to wipe the database completely or apply the schema changes manually, so this is a big improvement.

The way it works is as follows. I created a table in the database whose sole purpose is to store the schema version of the database. This is just an integer that starts at "1" and increments every time the schema changes. The source code also contains a version number, which is the schema version that the source code is programmed to use. When Sleet starts up, it compares the version number in the database with the version number in the source code to determine if the schema is out of date.

If the schema is out of date, it runs a series of migration scripts. Each migration script contains the SQL code necessary to migrate the database from one version to the next. For example, if the latest database schema version is "4", then the application will contain three migration scripts: 1-to-2, 2-to-3, and 3-to-4. By chaining these scripts together, the database schema can be updated no matter what version it currently is. For example, if the schema version of my database is "2", it will first execute the 2-to-3 migration script and then execute the 3-to-4 migration script. If it's "3", then it will just execute the 3-to-4 script. If it's "1", then it will execute all of them. All of this is done within a database transaction, so if something goes wrong during the migration process, the database will be restored to its previous state.

The psuedo-code below shows how this is done in code.

//connect to the database
Connection db = ...
db.setAutoCommit(false);

int schemaVersion = 4;
int curSchemaVersion = //"SELECT db_schema_version FROM sleet"
if (curSchemaVersion < schemaVersion) {
  //schema is outdated, run the migration script(s)
  Statment statement = db.createStatement();
  while (curSchemaVersion < schemaVersion) {
    String script = "migrate-" + curVersion + "-" + (curVersion + 1) + ".sql";
    SQLStatementReader in = new SQLStatementReader(new InputStreamReader(getClass().getResourceAsStream(script)));
    String sql;
    while ((sql = in.readStatement()) != null) {
      statement.execute(sql);
    }
    curSchemaVersion++;
  }

  //update the version number in the database
  //"UPDATE sleet SET db_schema_version = [schemaVersion]"

  //commit the transaction
  db.commit();
}

A couple weeks ago, I showed you how to send emails without an email client. In this blog post, I'm going to show you how to do the opposite--how to retrieve emails from an email server without an email client. As before, everything will be done on the command-line.

To do this, I'll be using the POP3 protocol. POP3 stands for Post Office Protocol version 3. Its purpose is to retrieve emails from an email server (like picking up mail from the post office, hence the name). The other popular email retreival protocol, which you may have heard of, is IMAP. IMAP offers many more features, like the ability to organize emails into folders, but as a consequence, it is more complex. POP3's feature-set is limited to just retrieving and deleting messages, so it's a lot simpler.

POP3 is similar to SMTP in that client/server communication is text-based. However, POP3 is a little simpler because the responses from the server do not have a plethora of numeric status codes. In POP3, there are only two responses: success responses (which begin with +OK) and failure responses (which begin with -ERR).

Connecting to a POP3 server

So let me show you how it works. I will be opening a POP3 connection to my Gmail account. Gmail requires that POP3 transactions be encrypted, so I can't use telnet like I did in my previous SMTP demo. The openssl command will allow me to open a sort of "encrypted telnet" connection.

openssl s_client -connect pop.gmail.com:995 -crlf -ign_eof

For more information on this command, read its man page: man s_client

If the POP3 server is not encrypted, the telnet command will work:

telnet pop3.server.com 110

Note: Most webmail services support POP3. You should be able to find the POP3 URL of your webmail service in your webmail's configuration settings or help pages.

POP3 commands

First, you obviously must authenticate yourself. There are many ways to perform authentication in POP3, but the simplest way is by using the USER and PASS commands. These allow you to enter your username and password directly.

+OK Gpop ready for requests from 68.80.246.118 cn9pf9267980vdc.5
USER mike.angstadt
+OK send PASS
PASS secret
+OK Welcome.

Now that I'm authenticated, I can start retrieving emails. The LIST command returns a list of all of my emails. Each email has an ID (the first number on each line) which is used to retrieve and delete individual emails. Note that these IDs can change with every POP3 session, so do not consider these to be permanent IDs! For instance, if you deleted one or more emails in a previous session, then the IDs of all the emails below the deleted emails will change in your next POP3 session. The number to the right of the ID is the size of the email in bytes. With Gmail, only the first 300 or so messages are shown for some reason. If anyone knows how to get the rest, leave a comment!

LIST
+OK 305 messages (57774596 bytes)
1 2017
2 3751155
3 10873184
...
305 3021

The STAT command is handy to have. It shows the total number of emails (the first number) as well as the total size in bytes of all the emails combined (the second number).

STAT
+OK 305 57791921

The RETR command retrieves an email, including both the headers and the body. The syntax is RETR <num> where <num> is the message ID. For example, to retrieve the fifth email, I would type RETR 5.

RETR 5
+OK message follows
Date: Mon, 17 Jan 2005 17:20:17 -0500
From: Mike Angstadt <mike.angstadt@gmail.com>
To: John Doe <jdoe@yahoo.com>
Subject: Hello John
...more headers...

Dear John,

How are you doing?

-Mike
.

!! Warning !!: Even though I have my Gmail POP3 access configured so that emails are not deleted when they are retrieved, it seems to be doing that anyway and I don't know why! Make sure that your email account is configured so that when you retrieve an email with POP3, it is not deleted from your inbox.

The DELE command, you guessed it, deletes an email. Like RETR, the syntax is DELE <num> where <num> is the message number. Note that the email won't actually be deleted until you terminate the POP3 session with the QUIT command. If your connection terminates unexpectedly, your emails will NOT be deleted.

DELE 1
+OK marked for deletion

If you mark an email for deletion by mistake, you can use the RSET command to undo it. This command will unmark all messages that have been marked for deletion. This means that when QUIT is sent, the emails won't be deleted.

RSET
+OK

And, as was already mentioned, the QUIT command closes the POP3 session, deleting all messages that were marked for deletion with DELE.

QUIT
+OK Farewell.

For more information on POP3, check out its specification document: RFC-1939

Also check out the SMTP server that I wrote, which supports POP3: https://github.com/mangstadt/Sleet

This post describes what I've learned during the second half of the Philly Emerging Technologies for the Enterprise Conference. See my previous blog post for a description of the first day. It was a great conference and I had a blast!

Keynote Address - Emerging Programming Languages

by Alex Payne

Alex started his talk by repeating the most common complaint people have about new languages--"why do we need another programming language?" His answer? Because evolution is a process that's constantly in motion--there's no way of knowing where the "jumping off point" is. As he gave this answer, a picture showing the evolution of the human skull was displayed behind him, implying that we are the result of a similar, albeit slower, kind of change (biological evolution).

When learning about new languages, which Alex does as a hobby, Alex's end goal isn't necessary to use the language, but to learn about the language's unique features and to try to incorporate those features into his work. One language he gave as an example had a certain elegant way of working with WSDLs which compelled him to implement a similar feature into one of his projects.

Alex described around two dozen very obscure programming languages, only 2 of which I've ever heard of (Go and CoffeeScript). He divided the languages up into categories, such as "Web Development", "Dynamic Programming", and "Querying Data".

Behind the scenes of Spring Batch

by Josh Long

Spring Batch is a Spring module that makes creating batch processes more standardized and less error prone. You basically define your job in an XML file. Then, using a combination of custom Java code and classes from the Spring Batch API, you write the logic of your batch job. It streams the batch data by reading the individual entry elements from the input data one by one and then writing out the processed data in chunks (e.g. 10 entries at a time). Because of this, you don't have to worry about getting OutOfMemory errors when processing large amounts of data.

I also thought it was cool that you can schedule your job to run on a regular basis by giving it a cron expression. In addition, you can have it generate a small web app that allows you to view the status of your jobs from the browser.

One thing that took me a little by surprise is that Spring Batch requires a connection to a database. It uses this database basically for logging purposes, like keeping track of the times the job ran and recording the errors that occurred (if any) while the job was running.

Spring Batch looks like a very clean and robust way of working with batch jobs. I definitely want to look more into it.

Dependency Injection Without the Gymnastics - Functional Programming Applied

by Runar Bjarnason and Tony Morris

This presentation was pretty unique in that the speaker, Tony, gave his talk via Skype from Australia! Runar was there in person and acted as the technician and the intermediary between the audience and Tony. It was about how to do dependency injection in Scala without having to resort to confusing XML files like with Spring.

The CoffeeScript Edge

by Trevor Burnham

Trevor explained some of the benefits that CoffeeScript brings to the table in this presentation. For example, following one of Douglas Crockford's words of wisdom, the Javascript code that CoffeeScript generates will never use the "==" operator. When comparing two variables in CoffeeScript, the syntax "x is y" is used, which translates to "x === y" in Javascript.

CoffeeScript also supports string interpolation, which allows you to concatenate strings using a cleaner syntax. For example:

dog = 'Spot'
x = "See #{dog}. See #{dog} run."

Another nice perk in CoffeeScript is that you don't have to separate array elements with commas if they are on separate lines. For example:

arr = [
  'One'
  'Two'
  'Three'
]

You can also use the @ operator as shorthand for this.

Trevor also made an interesting point about the increasing popularity of Javascript. Due to the increased usage of Javascript on the web, all the major browser makers (Microsoft, Google, Mozilla, Apple, and Opera) have been pouring money into making the language faster on their browsers. It's quite possible that no language in the history of computing has ever received this much financial backing.

JavaScript Testing: Completing the BDD Circle in Web Development

by Trevor Lalish-Menagh

This talk focused on how to write unit tests for Javascript code. Trevor did some live coding using some pretty impressive vim-foo, showing how to unit test Javascript code using the Jasmine framework. An important concept that he discussed was "spying" on functions. I'm not sure if this is unique to Jasmine, but it allows you determine whether a particular function was called or not, something that's very helpful in unit testing. Trevor also showed that it's possible to integrate your Javascript unit tests into a Maven build script.

Effective Scala

by Joshua Suereth

Joshua's talk focused on providing fairly advanced tips for writing good Scala code. He stressed the importance of using the Scala REPL (an interactive interpreter) during development. The REPL should be used on a regular basis to experiment with unfamiliar libraries and test out snippets of code. He also stressed the importance of staying immutable. If your objects are immutable, then it means (1) they are thread-safe and (2) they are hash-safe. He says that you should write your interfaces in Java because the bytecode of Scala interfaces doesn't convert well back to Java.

Joshua talked in depth about what's called "implicit scope". This is a special scope that basically lets you insert whatever variables you want into it. If used properly, it can be very powerful. One example Joshua gave was using implicit scope to define a collection of "Encoder" classes which convert various objects to byte arrays. It's designed that so any object can be passed into an "Encoder.encode()" method. Then, using implicit scope, the method delegates the object to the appropriate "Encoder" implementation for further processing.

Today, I attended the first half of the Philly Emerging Technologies for the Enterprise Conference down in Center City. This was a very good conference and I look forward to attending the second half tomorrow! Here's what I took in from the talks I attended.

Keynote Address - Self Engineering

by Chad Fowler

The conference started with a surprise visit from the mayor of Philadelphia, Michael Nutter(!!). Following the mayor was Chad Fowler, who talked about applying software development principles to improving your own life. One interesting thing he discussed was what's called a "QFD" (quality function deployment) graph, which is a technique for converting non-quantifiable requirements into quantifiable requirements. The example he gave was making a good cookie. Customers might say that they want a cookie that "tastes good", "has good texture", and "is cheap". These are all valid requirements, but completely non-quantifiable! What exactly makes a cookie "taste good"? More sugar? More chocolate? How much more? A QFD helps to break these requirements down into hard numbers.

Javascript, Programming Style, and Your Brain

by Douglas Crockford

This is the guy that wrote the excellent book, "JavaScript: The Good Parts", which contains insightful techniques for writing good Javascript code. He's also the author of JSLint, an online tool that helps to improve Javascript code. His talk was about Javascript and what to avoid doing when coding in the language. For instance, you should never use the "with" statement because it acts in unpredictable ways under certain circumstances. He also suggests never using the "switch" statement, since it's easy for a programmer to forget to include the "break" keyword inside of a "case" block.

Also, he says you should always put your opening curly braces on the same line to the right instead of on the next line to the left. In most languages, this issue is simply a matter of programmer taste and does not effect the actual behavior of the program. But in Javascript, there's one situation where it does have consequences:

return {
  foo:'bar'
};

return
{
  foo:'bar'
};

These two return statements both seem to do the same thing--return an inline object. But in fact, only the top example does this! The reason is that, since semi-colons are optional, Javascript auto-inserts a semicolon after the return keyword in the bottom example, causing it to exit the function and return nothing. It completely ignores the object that's defined below it (it won't even throw an error message). So, if you always put your curly braces on the right, you'll never have to worry about this quirk.

Java EE in the Cloud

by Gordon Dickens

In his talk, Gordon compared and contrasted a number of cloud-based JavaEE services. These services allow you to quickly deploy JavaEE web applications to the Internet and customize what kind of back-end software you want to use. For example, one cloud service he demoed lets you choose what database and web container you want to use.

In response to hearing some buzz about Java 7 being "cloud ready", Gordon did a close investigation of the current source code of JavaEE 7. He couldn't find anything substantial that was really worthy of that description. He said that Oracle intends to release JavaEE 7 during the third quarter of this year no matter what, and that anything that doesn't make it into version 7 will be pushed back to version 8.

SQL? NoSQL? NewSQL?!? What's a Java developer to do?

by Chris Richardson

This talk was after lunch, so I was a little sleepy, but I did my best to pay attention. Chris compared and contrasted three next-generation databases: MongoDB, Apache Cassandra, and VaultDB.

MongoDB is a document-oriented, NoSQL database. Every record in the database is a JSON object. Queries are pretty straight-forward--just pass the database a JSON object that has what you're looking for in it. Inserting data into MongoDB is fast because you don't have to wait for a response from the server when you send it commands. However, a downside is that it doesn't support ACID (i.e. transactions) like relational databases do. It's used by a number of large companies, such as bit.ly.

Apache Cassandra is another NoSQL database. However, it is column-oriented, instead of document-oriented like MongoDB. This means that a Cassandra database is basically one big hash map. Each record has a key and a value. The key can be anything (it doesn't have to be a number) and the value can also be anything. Chris said that this database is good for logging purposes because it can quickly ingest data. Netflix and Facebook both use this database.

VaultDB is known as a NewSQL database. From what I gathered from Chris' talk, it's basically just a relational database that resides completely in memory. It writes the database to disk like once an hour or something so it can be recovered if it crashes. A downside is that the API it uses is proprietary and still a work in progress. It has limited JDBC support.

Chris gave a good piece of advice for startup companies that are having trouble deciding what kind of database to use. You might be tempted to use one of these next generation databases because, as a startup, you're starting from scratch and don't have to do any sort of migration work that an established company running a relational database would have to do. However, the advantages that NoSQL and NewSQL databases bring to the table--namely speed and scalability--aren't things you really need as a new business. Since you're a small company, you don't have very many customers, so neither speed nor scalability is really an issue. In fact, you could probably hit the ground running much faster with a relational database because its tools, software, and support are more mature.

How GitHub Works

by Scott Chacon

This talk was given by the CIO of GitHub, Scott Chacon. He described the workplace culture at GitHub.

• Trust your employees - Your employees want to do a good job. Define what your expectations are, and they'll likely exceed them. Don't micro-manage.
• No work hours - The traditional 9 to 5 work day is a relic from the industrial revolution long ago. Programming is largely a creative process and you can't effectively box it into a rigid schedule. If you're not being productive, then why are you at work? At GitHub, people work when they want to.
• Headphones - If you're "in the zone" working on a programming problem, it can be hard to return to the zone after being interrupted. The rule that they have at the GitHub office is that, if you're wearing headphones, no one can interrupt you no matter what. They can send you an IM or an email, but they can't physically approach you at your desk.
• The chat room is the office - Not all the employees are in a single building. Many are scattered all over the world, so they have a chat room that everyone uses for much of their communication.
• Saying "No" - Scott talked about the importance of creating a culture where it's OK to say "No". This means that when people propose new ideas, their feelings aren't hurt if the idea is turned down by the team. It's important to establish this in order to encourage people to speak their minds without the fear of rejection and also to prevent bad ideas from being put into place and harming the company.

The Evolution of CSS Layout: Through CSS 3 and Beyond

by Elika J. Etemad

Elika is a member of the W3C CSS Working Group, so it was interesting to get a "behind-the-scenes" look as to how these specifications evolve. Elika started by giving a brief history of CSS and then gave a preview as to what can be expected in the future. She says that standardizing the rules for how elements are positioned on the page is the most complicated part because of all the various layout algorithms that are involved.

As to their interaction with Micro$oft, she said that they are productive, contributing members to the standardization process. Their involvement was lackluster during the long rule of IE6, but improved with the release of IE7.

Before Elika joined as a full-time employee, she was a dedicated member of the mailing list and an avid submitter of browser bugs. After several years of involvement, they offered her a job! It just goes to show that when the W3C says they are open to input from the community, they really mean it!