Aug 19 2011

Een korte mega top 50 update

De lijst
Niet in spotify:
  • Somebody That I Used To Know van Gotye feat Kimbra W
  • Called Out In The Dark van Snow Patrol
  • someone like you van Adele
  • The Adventures of Rain Dance Maggie van Red Hot Chili Peppers
  • Cinema van Benny Benassi ft Gary Go
  • Set Fire To The Rain van Adele
  • Titanium van David Guetta & feat. Sia
Mijn favorieten deze week:
  • Something in the water van Brooke Fraser
  • Fool for you van Krystl
  • Jet Lag van Simple Plan
  • The Lazy Song van Bruno Mars
De herrie weggefilterd is(valt mee deze weel)
  • Loca People van – What the fuck

Bekijk voor de gein eens mijn site op #mce_temp_url#

May 04 2010

How to create a CentOS 5.4 Amazon AMI with java and tomcat

After some experimenting with Google App Engine. There is one thing that bothers me the most. I cannot run any background programs. So it thought it was time to check out the competition at Amazon ec2. I like centos and I wanted to create an ami which run on 5.4, contains a servlet engine (Apache Tomcat) and is able to receive http requests on port 80. There are a lot of resources available (herehereherehereherehere,here and here) but none of them did exactly what I want.

The script I wrote (below) does the following things:

  • Uses the 32 bit version of CentOS 5.4 (only for Small or High-CPU Medium instances)
  • a 5GB file system
  • reroute traffic from port 8080 to 80 (using iptables)
  • Install a xen compatible kernel
  • Prevent the re spawning of tty2 to 6
  • Install java
  • Install tomcat & auto start

If you have any remarks/comments, please let me know, than I can improve this script.

First I installed centos 5.4 in a VMware environment and from there I created the Amazon AMI.

#create a 5GB image:

dd if=/dev/zero of=centos54ami.sf bs=1M count=5120

#create a filesystem:

/sbin/mke2fs -F -j centos54ami.sf

#create mountpoint:

mkdir /mnt/ec2-fs

#mount on loopback:

mount -o loop centos54ami.sf /mnt/ec2-fs

#create devices:

mkdir /mnt/ec2-fs/dev
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x console
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x null
/sbin/MAKEDEV -d /mnt/ec2-fs/dev/ -x zero

#create etc:

mkdir /mnt/ec2-fs/etc

#Create a proc point for the image and mount it:

mkdir /mnt/ec2-fs/proc
mount -t proc none /mnt/ec2-fs/proc

#vi yum-xen.conf file (on the local filesystem/not on the mount)

vi yum-xen.conf

name=CentOS-5.4 – Base
#released updates
name=CentOS-5.4 – Updates
#packages used/produced in the build but not released
name=CentOS-5.4 – Addons
name=CentOS 5.4 Extras $releasever – $basearch

#Run the yum installer and install the Core group:

yum -c yum-xen.conf --installroot=/mnt/ec2-fs -y groupinstall Core

#setup network settings:

vi /mnt/ec2-fs/etc/sysconfig/network-scripts/ifcfg-eth0


#Turn on networking:

vi /mnt/ec2-fs/etc/sysconfig/network


#create resolv.conf:

vi /mnt/ec2-fs/etc/resolv.conf

search z-2.compute-1.internal?nameserver

#Set up the hard drives:

vi /mnt/ec2-fs/etc/fstab

/dev/sda1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/sda2 /mnt ext3 defaults 0 0
/dev/sda3 swap swap defaults 0 0

#Install wget, curl and iptables

yum -c yum-xen.conf --installroot=/mnt/ec2-fs -y install wget
yum -c yum-xen.conf --installroot=/mnt/ec2-fs -y install curl
yum -c yum-xen.conf --installroot=/mnt/ec2-fs -y install iptables

#disable selinux

vi /mnt/ec2-fs/etc/selinux/config


#comment out the respawning of tty2 to 6

vi /mnt/ec2-fs/etc/inittab

comment out mingetty tty2 to 6

#create script to retrieve the public key

vi /mnt/ec2-fs/usr/local/sbin/

if [ ! -d /root/.ssh ] ;
then mkdir -p /root/.ssh
chmod 700 /root/.ssh
# Fetch public key using HTTP
curl -f > /tmp/my-key
if [ $? -eq 0 ] ;
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm -f /tmp/my-key

#add execute rights

chmod +x /mnt/ec2-fs/usr/local/sbin/

#add line to rc.local:

vi /mnt/ec2-fs/etc/rc.local


#download & install java

mv <long filename> jre-6u18-linux-i586.bin
chmod +x jre-6u18-linux-i586.bin
mv jre1.6.0_18 /mnt/ec2-fs/usr/lib/

#download & install tomcat

tar -xzf apache-tomcat-5.5.28.tar.gz
mv apache-tomcat-5.5.28 /mnt/ec2-fs/opt/tomcat
echo "export CATALINA_HOME=/opt/tomcat" >> /mnt/ec2-fs/root/.bashrc
echo "CATALINA_BASE=/opt/tomcat"  >> /mnt/ec2-fs/root/.bashrc

#autostartup tomcat

vi /mnt/ec2-fs/etc/init.d/tomcat

# tomcat
# chkconfig: 2345 80 30
# description: 	Start up the Tomcat servlet engine.
# Source function library.
. /etc/init.d/functions
export JAVA_HOME="/usr/lib/jre"
case "$1" in
        if [ -f $CATALINA_HOME/bin/ ];
	    echo $"Starting Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/
        if [ -f $CATALINA_HOME/bin/ ];
	    echo $"Stopping Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/
 	echo $"Usage: $0 {start|stop}"
	exit 1
exit $RETVAL
chmod +x /mnt/ec2-fs/etc/init.d/tomcat
cd /mnt/ec2-fs/etc/rc5.d
ln -s /mnt/ec2-fs/etc/init.d/tomcat /mnt/ec2-fs/etc/rc5.d/S71tomcat

#Chroot and auto start the services sshd and tomcat:

/usr/sbin/chroot /mnt/ec2-fs /bin/sh
/sbin/chkconfig --level 345 sshd on
/sbin/chkconfig tomcat on

#create link for java lib

cd /usr/lib
ln -s jre1.6.0_18 jre

#set the correct java path

echo "export JAVA_HOME=/usr/lib/jre" >> /root/.bashrc
echo "export PATH=/usr/lib/jre/bin:$PATH" >> /root/.bashrc
/usr/sbin/useradd -d /opt/tomcat tomcat
chown -R tomcat:tomcat /opt/tomcat

#remove default installed applications

cd /opt/tomcat/webapps
rm -r balancer/
rm -r webdav/
rm -r tomcat-docs/
rm -r servlets-examples/
rm -r jsp-examples/

#install custom kernel

gunzip modules-2.6.16-ec2.tgz
tar -xvf modules-2.6.16-ec2.tar
/sbin/depmod -ae 2.6.16-xenU

#fix some Xen guest kernels things

vi /etc/

# This directive teaches ldconfig to search in nosegneg subdirectories
# and cache the DSOs there with extra bit 0 set in their hwcap match
# fields. In Xen guest kernels, the vDSO tells the dynamic linker to
# search in nosegneg subdirectories and to match this extra hwcap bit
# in the file.
hwcap 0 nosegneg

#create the necessary links and cache


#redirect port 80 to 8080

/sbin/iptables --flush
/sbin/iptables --table nat --flush
/sbin/iptables -t nat -A PREROUTING -j REDIRECT -p tcp --destination-port 80:80 --to-ports 8080

#Exit out of your chroot:



yum -c yum-xen.conf --installroot=/mnt/ec2-fs clean all

#Unmount the image

umount /mnt/ec2-fs/proc
umount -d /mnt/ec2-fs

#install ruby

yum install ruby

#Download the EC2 AMI Tools:


#Install the EC2 AMI Tools

rpm -Uvh ec2-ami-tools.noarch.rpm

#bundle the your image

/usr/local/bin/ec2-bundle-image -i centos54ami.sf -c <your certificate> -k <your private key> -u <your account number>

#upload to s3

/usr/local/bin/ec2-upload-bundle -b <your bucket name> -m /tmp/centos54ami.sf.manifest.xml -a <your aws access key> -s <you aws secret key>

With the AWS console, you can now register and startup the AMI

#log on to the newly created instance

ssh -i <your private key> <your public generated dns name>

Feb 16 2010

Uploading files from GAE/J to Amazon S3 with html posts

In this post I will explain how to post files from the Google App Engine to Amazon S3.

Google app engine is a nice platform to host your application, but it has a few limitations.
  • Data structures larger than 1MB are not supported.
  • Timeout per request 30 seconds.
Currently (if you’re not storing huge amounts of data > 50TB)  Amazon S3 is more expensive than Google. But if you need to store more data or larger  files or you already have a Amazon s3 bucket in place or whatever reason. It would be nice to use Amazon s3 for your storage.

For uploading files there are a few alternatives:

  1. Uploading using http post. Amazon described it here.
  2. Directly from java.
  3. Using a adobe flex solution.
  4. Other way (e.g. custom flash application/applet/java fx solution)
In this blog post I will explore the first method. The downside for this method is that you can only upload one file at a time. In a next post I will explore how to upload multiple files at the same time. Probably I will be using SWFUpload for that.
In yet another post I will explain how to upload directly from java. The downside for this is that you have to pay for bandwidth twice, once from the browser to GAE and once from GAE to Amazon S3. But I may be necessary in some cases.


First the html file for choosing a file and uploading it to Amazon:
<form action="http://<<YOUR BUCKET>>" method="post" enctype="multipart/form-data">
		Key: <input name="key" value="uploads/" /><br />
		<input type="hidden" name="acl" value="private" />
		<input type="hidden" name="AWSAccessKeyId" value="$awsaccesskeyid" />
		<input type="hidden" name="Policy" value="$policy" />
		<input type="hidden" name="Signature" value="$signature" />
	<input type="hidden" name="redirect" value="" >
		File: <input type="file" name="file" /> <br />
		<input type="submit" name="submit" value="Upload to Amazon S3" />
The html file has a few special things which will be filled in using a velocity template (see my previous post).


Needs to filled in with the Access key ID which can be found at the  security settings  at Amazon AWS Portal


Needs to be filled with a base64 encoded policy file, it could look a little like this. (the base64 encoding I explain later)
{"expiration": "2010-10-01T00:00:00Z",
 "conditions": [
  {"bucket": "<<BUCKET_NAME>>"},
  ["starts-with", "$key", "uploads/"],
  {"redirect": "http://<<YOUR SITENAME>>/ successful upload.html" },
  {"acl": "private"},
  ["content-length-range", 0, 1048576]
This policy says the following things:
  • The policy is valid until October the 10th 2010
  • The files must be uploaded to bucket <<BUCKET_NAME>>
  • The file should be uploaded to a path that starts with uploads/
  • After a successful upload, Amazon should redirect you to http://<<YOUR SITENAME>>/ successful upload.html
  • After uploading nobody can access the file, check here for more information
  • The maximum size of the file is 1MB


The signature that was created when signing the policy with you Secret Access Key. The signing part is explained on the end of this post.

Http Servlet

The servlet that creates the html file looks like this:
public class UploadPost extends HttpServlet{
	public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
		PrintWriter out = resp.getWriter();
		VelocityContext context = new VelocityContext();
		context.put("awsaccesskeyid","<<YOUR AWS ACCESSKEY>>");
		context.put("policy", "<<YOUR BASE64 ENCODED POLICY FILE ");
		context.put("signature","<<YOUR SIGNATURE>>>");
		StringWriter writer = new StringWriter();
		Template template = VelocityEngineManager.getTemplate("upload_post.html");
		template.merge( context, writer );
In the java code you should place your own awsaccesskeyid, policy and signature.
The post is only an example of the possibilities, some things you may want to consider:
  • Generate policies on the fly server side with a shorter expiration date.
  • Create a unique filenames for the user.

Generate base64 encoded policy files and signatures

For sending files to Amazon S3, a policy and a signature is required. The following code can create a base64 encoded policy string and signature for you.
import org.apache.commons.codec.binary.Base64;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
public class Main {
	static String aws_secret_key = "<<YOUR SECRET AWS KEY>>";
	public static void main(String[] args) {
		String policy_document="";
		try {
			policy_document = readFileAsString("policy.txt");
		} catch (IOException e1) {
			System.err.print("Could not open policy file");
		try {
			String  policy = Base64.encodeBase64String(policy_document.getBytes("UTF-8")).replaceAll("\n","").replaceAll("\r","");
			Mac hmac = Mac.getInstance("HmacSHA1");
			hmac.init(new SecretKeySpec(aws_secret_key.getBytes("UTF-8"), "HmacSHA1"));
			String signature = Base64.encodeBase64String(hmac.doFinal(policy.getBytes("UTF-8"))).replaceAll("\n", "");
		} catch (UnsupportedEncodingException e) {
			// TODO Auto-generated catch block
		} catch (NoSuchAlgorithmException e) {
			// TODO Auto-generated catch block
		} catch (InvalidKeyException e) {
			// TODO Auto-generated catch block
	private static String readFileAsString(String filePath) throws{
		StringBuffer fileData = new StringBuffer(1000);
        BufferedReader reader = new BufferedReader(new FileReader(filePath));
        char[] buf = new char[1024];
        int numRead=0;
        while(( != -1){
            String readData = String.valueOf(buf, 0, numRead);
            buf = new char[1024];
        return fileData.toString();
Feb 12 2010

Combining GAE, Apache Velocity and jQuery

If some of the earlier posts I used jQuery ajax posts to change the appearance of the user page. In this blog post I’m using Velocity to generate the html pages server-side.

Getting velocity up and running
To get started I placed the velocity-1.6.3.jar in the WEB-INF\lib folder.
The first try of getting everything up and running was not successful. GAE does not allow to load templates with the out of the box available resource loaders. So I had to create my own. Most of the code I got from here.

public class velocityResourceLoader extends ResourceLoader {
	private Vector paths = null;
	public long getLastModified(Resource arg0) {
		return arg0.getLastModified();
	public InputStream getResourceStream(String template) throws ResourceNotFoundException {
		int size = paths.size();
		for (int i = 0; i < size; i++) {
			String path = paths.get(i);
			InputStream is = null;
			try {
				is = new FileInputStream(path + "/" + template);
				return is;
			} catch (FileNotFoundException e) {
				// skip
		throw new ResourceNotFoundException(template);
	public void init(ExtendedProperties configuration) {
		paths = configuration.getVector("path");
	public boolean isSourceModified(Resource arg0) {
		return false;

The method getResourceStream takes the argument template and tries to find the template in the path variable, which must be initialized using a property.

Initializing Velocity

Next I created a singleton to set up and store the VelocityEngine. In this singleton I also added a function to find a template.

public final class VelocityEngineManager {
	public static final Logger log = Logger.getLogger(VelocityEngineManager.class.getName());
	static VelocityEngine engine = new VelocityEngine();
	static boolean init = false;
	public static void init() {
		if (!init) {
			// engine = new VelocityEngine();
			engine.setProperty("resource.loader", "file");
			engine.setProperty("file.resource.loader.path", "templates");
			engine.setProperty("runtime.log.logsystem.log4j.logger", log.getName());
			try {
				init = true;
			} catch (Exception e) {
				// TODO Auto-generated catch block
	public static Template getTemplate(String template) {
		try {
			return engine.getTemplate(template);
		} catch (ResourceNotFoundException e) {
		} catch (ParseErrorException e) {
		} catch (Exception e) {
		return null;

In the servlet the following code is needed to load a template, determine if a user is signed in and based on that choose a template to load.

using Velocity templates

public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
	PrintWriter out = resp.getWriter();
	HttpSession session = req.getSession(true);
	User user = (User) session.getAttribute("user");
	VelocityContext context = new VelocityContext();
	Template t = new Template();
	if ( user!=null) {
		t = VelocityEngineManager.getTemplate("main_newuser.html");
		context.put("username", user.getFullName());
	} else {
		t = VelocityEngineManager.getTemplate( "main_knownuser.html" );
		context.put("username", "No user logged on!");
	StringWriter writer = new StringWriter();
	t.merge( context, writer );

The templates are written in plain HTML + javascript with two differences

1) The velocity parser doesn’t like it when you use $. combinations in the template. There is already a bug reported for this , but the version I’m using doesn’t contain the fix yet. A workaround is to replace all $ tags with jQuery. So use “jQuery.ajax({” instead of “$.ajax({“.

2) Instead of retrieving dynamic information with ajax, the $ tags are used to set data service side. So the part in the html template showing the username looks like:

<div id="userfullname" class="outputTextArea">$username</div>

Other thoughts

The advantage of using this method are:

  • That you need less http gets/posts to the server.
  • The performance seems better, but that can also be because of my lacking javascript skills.
  • The html files are smaller, less complex, because a lot can be done server-side.
  • Better separation of code and markup.

The disadvantages are:

  • There is more CPU load on the server.
  • There may be some limitations in velocity (maybe using FreeMarker can solve that)
  • Not sure if caching can be a problem.
  • The page completely needs to be reloaded in case of a change (but of course ajax call still be used if necessary)
Feb 10 2010

Using the memcache with GAE/J: Part 1

The last time I wrote about time-outs. One way of preventing them is using the memcache. The memcache can also be used for improving the performance.

In this blog post I will extend the user session examples with memcache functionality. If for some reason the memcache does not function or is cleaned. The http session is used as a fallback scenario.

To identify which memcache information will be retrieved, a cookie needs to be stored at the client site.

Accessing the memcache

First I created a singleton to store the memcache and to access the cache from.

public final class SessionMemCacheManager {
	private static CacheFactory cacheFactory;
	private static Cache cache;
	public static final Logger log = Logger.getLogger(SessionMemCacheManager.class.getName());
	public static Cache getCache() {
		if (cache==null){
			try {
				cacheFactory = CacheManager.getInstance().getCacheFactory();
				return cacheFactory.createCache(Collections.emptyMap());
			} catch (CacheException e) {
				return null;
		}else {
			return cache;

The function getCache can be called to access the memcache.

Storing information in the memcache and creating cookies.

The signin servlet I extended with the following code.

String sessionid = ServletUtilities.getCookieValue(req.getCookies(), "sessionid",ServletUtilities.generateSessionId());
Cache cache = SessionMemCacheManager.getCache();
cache.put(sessionid, user);
resp.addCookie(new Cookie("sessionid", sessionid));

The first line retrieves the sessionid from the cookie or will generate a new sessionid.

The second line gets the cache from the singleton.

The third line put the sessionid combined with the sessionid in the memcache.

And the forth line adds a cookie (with the sessionid) at the client site.

In the first line a few new functions are used. The first one (getCookieValue) I stole from here.

The second one I created myself.

public static String generateSessionId() {
	final UUID idOne = UUID.randomUUID();
	try {
	} catch (final UnsupportedEncodingException e) {
		// TODO Auto-generated catch block
	return null;

Retrieving information from the memcache

The servlet “to verify if a use was signed” in is also modified

String sessionid = ServletUtilities.getCookieValue(req.getCookies(), "sessionid",ServletUtilities.generateSessionId());
resp.addCookie(new Cookie("sessionid", sessionid)); //This does nothing if the cookies already exists
Cache cache = SessionMemCacheManager.getCache();
User user = (User) cache.get(sessionid);
if ( user==null) { //only if the memcache doesn't contain a user
	HttpSession session = req.getSession(true);
	user = (User) session.getAttribute("user");

The third line tries to retrieve the User object from the memcache.

If this didn’t resulted in a valid user object, the user is retrieved from http session.

Deleting information from the memcache

The singout servlet is extended with the following lines

String sessionid = ServletUtilities.getCookieValue(req.getCookies(), "sessionid",null);
if (sessionid!=null){
	Cache cache = SessionMemCacheManager.getCache();

These lines will removed the cached entry from the memcache

Open point

I have no idea how long the memcache will remain available. After some testing I found that deploying a new application does not affect the memcache. Also Google advices to also persist the information if the data is “suddenly not available”. For now I only use the memcache for sessions. So I think that should be enough.

Feb 02 2010

Datastore time-outs with GAE/J: Part 1

UPDATE feb 11 2010 : Apparently Google did some changes in de SDK version 1.3.1, that will make sure that if a timeout occurs the transaction is retried automatically.

Most of the content in this blog post probably is now redundant .

In my last blog entry I wrote about using sessions with Google App Engine.

There I found out that I sometimes receive an DatastoreTimeoutException.

I think that there are a few ways to handle with these time-outs.

  1. Catch the exception and retry, probably with a maximum of 7 to 8 times, because of the web request timeout of 30 seconds
  2. Use the memcache
  3. Create a task queue

The first one I will explore in this blog post. The memcache and task queue I will maybe discuss a next time.

Apparently I’m not the only one with the time-out exception at stackoverflow it is also mentioned. Also at Google there is an article about it

The good news is that there is a solution for the catching approach. The bad new is, that it is written in Python.

The timeout error occurred in two locations in my application

  • session.invalidate(); (while signing out a user)
  • pm.makePersistent(user); (while creating a new user)

A solution for retrying the second line could look a bit like this:

User user = new User(email, fullName, password);
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
	int timeout_ms = 100;
	int retrycount =0;
	while(true) {
		try {
		catch (DatastoreTimeoutException e) {
			if (retrycount>6){
				throw e;
			timeout_ms *= 2;
			try {
			} catch (InterruptedException e1) {
} finally {

But that seems a lot of boilerplate code that needs to be put in every time I want to access the datastore. The original code is below.

User user = new User(email, fullName, password);
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
} finally {

So this is not a solution yet, I will be getting back on this.

Jan 30 2010

Using sessions with Google App Engine

Today I’m gonna make sure the users who created an account using my functionality described in the previous blog can actually log on and that a http session is created. This session will remain valid until the user signs out or if the user closes the browser.

For that I create a new html page. This page should contain a few things.

If a user is not signed in:

  • A sign in link
  • A text stating that no user in signed in

If a user is signed in:

  • A sign out link
  • The full name of the user

At the sign in page the user can enter his e-mail address and password. If the password is correct they will be automatically redirected to the initial page.

If the username/password combination is not correct, the user must be redirected to the “wrong password” page.

To do all this, I make use of the jQuery JavaScript library. The downloaded jquery-1.4.1.js file I placed in the /war/js folder in the Eclipse project.

To enable sessions in GAE the appengine-web.xml needs to be adjusted, the following needs to added.


Checking for sign in users

First I have to create a servlet to check if there is a session and if it contains a valid user object in the user attribute. The result of this is returned in xml. The browser will process this information with jQuery.

public class GetUserServlet extends HttpServlet {
	public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
		HttpSession session = req.getSession(true);
		PrintWriter out = resp.getWriter();
		User user = (User) session.getAttribute("user");
		if ( user!=null) {
			out.println("" + user.getFullName() + "true");
		} else {

The body for the session_example.html file is very simple.

	<form action="/signin" method="post">
		<p>Email: <input name="email"></p>
		<p>Password: <input name="password" type="password"></p>
		<p><input type="hidden" name="passwordOK" value="/session_example.html"></p>
		<p><input type="hidden" name="passwordNOK" value="/password_error.html"></p>
		<p><input type="submit" value="Sign in"></p>

To make sure that the div with the userfullname id contains the correct value (provided by the servlet in the xml tag user) a piece of jQuery JavaScript is required in the session_example.html file.

$(document).ready(function() {
		type: "POST",
		url: 'checkuser',
		success: function(xml) {
		authenticated = $("authenticated", xml).text();
		if (authenticated == "true") {
			$("#userfullname").html($("name", xml).text());
			$("#userfullname").html("No user signed in!");

When the html page loads, JavaScript executes a jQuery ajax post to the servlet mapped at checkuser, if the post was successful, the returned xml is parsed.

If the authenticated value is equal to true, the div with the id userfullname is filled the value of the user attribute from the xml and the sign in link is made hidden.

If authenticated not is true, the userfullname is filled with a fixed value “No user signed in!” and the signout link is made hidden.

This all takes care of checking is a user is signed in. Of course the user also needs to sign in.

Signing in users.

When a user clicks the sign link, the signin.html file opened.

	<form action="/signin" method="post">
		<p>Email: <input name="email"></p>
		<p>Password: <input name="password" type="password"></p>
		<p><input type="hidden" name="passwordOK" value="/session_example.html"></p>
		<p><input type="hidden" name="passwordNOK" value="/password_error.html"></p>
		<p><input type="submit" value="Sign in"></p>

If the “Sign in” button is pressed, a post is done to the servlet mapped to signin, besides the email and password also two hidden parameters are send. These contain the urls where the servlet should redirect to if the password is correct or incorrect.

The signin servlet processes the post.

public class SingInServlet extends HttpServlet {
	public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
		String emailParam = req.getParameter("email");
		String passwordParam = req.getParameter("password");
		String passwordOKredirectParam = req.getParameter("passwordOK");
		String passwordNOKredirectParam = req.getParameter("passwordNOK");
		HttpSession session = req.getSession(true);
		User user = null;
		PersistenceManager pm = PMF.get().getPersistenceManager();
		try {
			Query query = pm.newQuery(User.class);
			query.setFilter("email == emailParam");
			query.declareParameters("String emailParam");
			try {
				List users = (List) query.execute(emailParam);
				for (User resultuser : users) {
					user = resultuser;
				if (user!=null){
					if (user.authenticate(passwordParam)){
						session.setAttribute(session.getId(), user);
					} else{
				} else {
			finally {
		finally {

The servlet does a few things:

  • Read all the parameters
  • Creates a new session, if not one already exists
  • Starts a jdo query to the datastore, based on the provided email address
  • If the user can be authenticated, the user object is stored as a session attribute.
  • And finally the user is redirected to the session_example.html or password_error.html page.

Signing out

Almost everything is done, signing out is the only thing left. And just for fun, I’m doing that a little different.

<div id ="signout"><a href="signout">Sign Out</a></div>

If a user clicks the “Sign Out” link, the signout link should be opened, but with some jQuery I will override the click function.

	$(document).ready(function() {
		$("#signout").click(function(e) {
				type: "POST",
				url: 'signout',
				success: function(data) {
					window.location.href = "/session_example.html";

When a user now clicks the sign out link, a post is done to the signout servlet, and after that the user is redirected to the same page, causing the page to reload and detect that the user has logged out.
The default action (going to the signout link is disabled), making sure that that action is not executed.

The sign out servlet is pretty straightforward, it checks if a session exists and if it does, the session is invalidated.

public class SignOutServlet extends HttpServlet {
	public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
		HttpSession session = req.getSession(false);
		if (session!=null){


While testing online I found out that I missed a thing in my previous blog. The logging at the appengine site was showing errors like


I fixed that by adding “implements” to the user class.

Hmmm, it seems like I don’t complete have it yet. Locally it is running fine, but online at, the performance is bad.

I’m receiving all kind of the following errors:

Uncaught exception from servlet Unknown at

UPDATE feb 11 2010 : Apparently Google did some changes in de SDK version 1.3.1, that will make sure that if a timeout occurs the transaction is retried automatically.

Next steps

This sign in functionality is nice, it does the trick, but is not really nice looking and is missing some functionality. In the near future I will be adding some functionality:

  • Taking a look at the performance issue
  • Some layout, perhaps following Web Form Design Patterns: Sign-Up Forms
  • Pop up sign in e.g. like twitter currently uses.
  • Signing in using https
  • Using an OpenID to log on
  • Add forget password/generate new password functionality
  • Adding cookies, so that if a user closes the browser and comes back later, the site still knows who he is.
Jan 27 2010

Creating persistence users

Today I would like to test something with persistence. The idea is that I will create an user class, put some annotations in there, so that it is persistence. After that create an html form that will put some data in the database using a servlet.

For the user class I want to keep it as simple as possible, but safe. I only want to have an email, full name and a password in the class.

The password needs to be stored as an hash in the database. I learned that it is safe to also add some random salt to the hash, so that is not so easy rainbow hack. So I also need to add a salt attribute and password hash attribute to the class.

The information should be sent over a https connection, but I will not try this yet. I will be adding checking for a valid or already existing email address later in a different post. Also confirm messages will be added later, so it can be checked that the email address actual exists and belongs to the user adding the user. Support for Captcha will also be part of a different post.

In the java code I to try to be compliance with the Code Conventions for the Java from Sun.

The User class
First we need a class for the user.

package jvdkamp.example.userexample.domain;
 * Used for storing Users
 * @author Jurgen van de Kamp
public class User {
	private String email;
	private String fullName;
	private String passwordHash; //the base64 encoded SHA1 hashed password + added salt
	private String passwordSalt; // the random salt added to the password
	 * Creates a user object
	 * The provided password salted, SHA1 hashed and base64 encoded
	 * The password will be stored as passwordHash and passwordSalt
	 * @param email the email address of the user
	 * @param fullName the fullName of the user
	 * @param password the not encrypted user pass password
	public User(String email, String fullName, String password) { = email;
		this.fullName = fullName;
		this.passwordHash = ""; // TODO needs to be generated
		this.passwordSalt = ""; // TODO needs to be generated
	 * @return email
	public String getEmail() {
		return email;
	 * @param email
	public void setEmail(String email) { = email;
	 * @return fullName
	public String getFullName() {
		return fullName;
	 * @param fullName set the fullName of the user. including first, middle and last name
	public void setFullName(String fullName) {
		this.fullName = fullName;

To be able to make this class persistent, it needs to be annotated.

With the App Engine you have two possibilities JDO and JDA. Google uses DataNucleus to enhance the classes so they can be stored in the Datastore. I will be using JDO.
For make a class persistence an annotation should be added to the class.

@PersistenceCapable(identityType = IdentityType.APPLICATION).

Because the identityType is APPLICATION, we have to specify how the primary key for this class looks like.
Google provided the class for that. So we will also add an attribute key to the class. The attribute key also needs to be annotated with the @PrimaryKey annotation because it is the primary key.

Every attribute we want to store in the datastore needs to be annotated with @Persistent. The new key attribute needs something extra “valueStrategy = IdGeneratorStrategy.IDENTITY” to make sure that the value for the primary key is automatically generated.

The modified class looks like this, the added lines are marked.

import javax.jdo.annotations.IdGeneratorStrategy;import javax.jdo.annotations.IdentityType;import javax.jdo.annotations.PersistenceCapable;import javax.jdo.annotations.Persistent;import javax.jdo.annotations.PrimaryKey; 
@PersistenceCapable(identityType = IdentityType.APPLICATION) 
public class User {
	@PrimaryKey	@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)	private Key key;
	@Persistent	private String email;
	@Persistent	private String fullName;
	@Persistent	private String passwordHash; // the base64 encoded SHA1 hashed password + added salt
	@Persistent	private String passwordSalt; // the random salt added to the password
	//rest of the class left out

The objects in this class are now ready to be stored in the datastore. The class still misses two things. The password is not hashed yet. And in the future, when we are using the class to also authenticate the users, a function needs to be added for that.

The most code I took over from the Open Web Application Security Project OWASP. I will not discuss it in detail in this post.
Because we need to base64 encode a string and Google App Engine does not provide a function for that, I downloaded a lib from Apache Commons and put the jar in the /war/WEB-INF/lib folder of my project.

Function to create a password hash and random salt

public PasswordAndSalt(String password) throws NoSuchAlgorithmException, UnsupportedEncodingException {
	byte[] bSalt = new byte[8];
	SecureRandom random = SecureRandom.getInstance("SHA1PRNG");
	byte[] bDigest = getHash(password,bSalt);
	this.passwordHash = Base64.encodeBase64String(bDigest);
	this.passwordSalt = Base64.encodeBase64String(bSalt);

Function to create a SHA-1 hash.

private byte[] getHash(String password, byte[] salt) throws NoSuchAlgorithmException, UnsupportedEncodingException {
	MessageDigest digest = MessageDigest.getInstance("SHA-1");
	byte[] input = digest.digest(password.getBytes("UTF-8"));
	input = digest.digest(input);
	return input;

I added code to the User Class constructer to store the password hash and salt hash.

PasswordAndSalt passwordAndSalt = new PasswordAndSalt(password);
this.passwordHash = passwordAndSalt.getPasswordHash();
this.passwordSalt = passwordAndSalt.getPasswordSalt();

A function to compare an unencrypted password with the stored password. In this blog post I don’t use this function yet.

public boolean authenticate(String password) {
	byte[] bDigest = Base64.decodeBase64(this.passwordHash);
	byte[] bSalt = Base64.decodeBase64(this.passwordSalt);
	byte[] proposedDigest = null;
	try {
		proposedDigest = getHash(password, bSalt);
	} catch (NoSuchAlgorithmException e) {
		e.printStackTrace(); //TODO add some log information
		return false;
	} catch (UnsupportedEncodingException e) {
		e.printStackTrace(); //TODO add some log information
		return false;
	return Arrays.equals(proposedDigest, bDigest);


Now, after completion of the user class, a user also needs to enter this information on the site, for this I created an HTML file, and placed that in the war folder of the project.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "<a href=""></a>">
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <title>Create USer</title>
  <form action="/createUser" method="post">
    <p>Full Name <input name="fullName"></p>
    <p>Email: <input name="email"></p>
    <p>Password: <input name="password" type="password"></p>
    <p><input type="submit" value="Create User"></p>

Servlet for processing the post

If the user presses the button “Create User”, a post is done to /createUser url on the site, but there is no servlet running to capture the post, so we have to create one. The servlet retrieves the posted fullName, email and password and then creates an User object.

package jvdkamp.example.userexample.servlets;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import jvdkamp.example.userexample.domain.User;
public class CreateuserServlet extends HttpServlet {
	public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
		String fullName = req.getParameter("fullName");
		String email = req.getParameter("email ");
		String password = req.getParameter("password");
		try {
			User user = new User(email, fullName, password);
		} catch (NoSuchAlgorithmException e) {

We also have to tell the servlet engine that we have a new servlet running. The following lines need to added to the web.xml in the /war/WEB-INF folder.


After all this coding we can test if our newly “create user” page is working. Go to http://localhost:8888/createuser.html. Tun the app and create a user. After pressing the button a white page is the result. If we take a look at the datastore viewer(http://localhost:8888/_ah/admin/datastore) to see if the post did have a result, we see nothing.

This is because I forget one thing. In the userservlet I created a user object, but I did not make it persistence. This needs to be done by calling a makePersistent function in a Persistence Manager Factory. We need to create such a factory, Google explains here something about that.

Persistance Manager Factory

Added code to the CreateuserServlet class

package jvdkamp.example.userexample.helpers;
import javax.jdo.JDOHelper;
import javax.jdo.PersistenceManagerFactory;
public final class PMF {
	private static final PersistenceManagerFactory pmfInstance =
	private PMF() {}
	public static PersistenceManagerFactory get() {
		return pmfInstance;

If we now restart and go back to the create use htm pagepage and retry creating an user. We have a result in the datastore.

Next time I’ll try to use the newly created functionality to add the possibility to log on and track the user during its browsing session.

Jan 26 2010

First Google App Engine Application

In this post I will explain the steps I followed to create my first Google App Engine Application.
For creating a new application I first installed the following software:

Create the first application

  • Create a new “Web Application Project”.
  • Choose a Project name e.g. HelloWorld
  • Choose a Package e.g. jvdkamp.example.HelloWorld
  • Deselect “Use Google Web Toolkit”
  • And create the project.
  • Now press the “Run” Button
  • Go to http://localhost:8888/ with your favorite browser and click on the HelloWorld hyperlink
  • Wow the first Google App Engine application
Put the application online
Off course I don’t want to have this nice application just for myself, so now it is time to upload it to the
  • First you need to sign up for an App Engine Account at
  • From there you can create a new application mine is jvdkamp-blog
  • Then go back to Eclipse, press the button “Deploy App Engine Project”
  • Fill in you Google credentials and the freshly Generated Application Identifier and deploy
  • And now your application is available to the general public on http://<Application Identifier>

Putting in a favicon.ico

When I logged on to my application on http://localhost:8888/, I noticed that I saw a warning

WARNING: No file found for: /favicon.ico
I don’t like warnings, so to prevent the warning I created a new favicon.ico here. and placed the .ico file in the war folder in Eclipse and now I have a nice icon in the tab when I browse to my site and offcourse no more warnings.

Well this was very basic. Next post I will try to do something with persistence (saving something in the database).

Jan 26 2010

First Post

Well, this is new for me. In the coming time I will try to keep this blog up to date with my experience using the Google App Engine.

I will post some code examples, links to sites I used for research and last but not least also learn something from the comments I will hopefully receive.

The things on my shortlist are:

  • Getting Google App Engine Eclipse Plug-in up and running
  • First project: Creating user accounts
  • Maintaining sessions cookies etc.
  • Etc.

WordPress Themes