Ideaboard Technical Manual
(mak1g11, mrt1g11)

Architecture

We split our application into two basic parts: the front-end and the back-end. We decided to implement a client-server architecture, where the client uses a RESTful API to communicate with the server. The back-end would serve as a RESTful API for our front-end, returning JSON objects to be manipulated on the web page. The RESTful API server would also need to be able to carry out POST, PUT and DELETE functions in addition to GET the functions. We decided on this because:

  • We wanted the the users to be able to access their notes from anywhere.
  • A RESTful API back-end is de-coupled from the front-end implementation, allowing any sort of front-end implementation. This is important as in the future we may build a native mobile application which can use the same back-end as the web application.
  • RESTful APIs are much more simple to interact with than alternatives such as SOAP.

With these thoughts in mind, we set about making creating our server.

Back-end

Technologies

The main technology we decided to use to create our back-end was Node.js . This was chosen over a number of technologies such as the MVC ASP.NET and Java Play Framework . These two focused a lot more work on server-side and were too heavy-weight for a simple RESTful API. Moreover, Node.js was a technology we had to work on in our Group Development Project so we viewed using this as a perfect opportunity to learn a new technology while killing two birds with one stone.

Model

We designed three main structures: Boards, Columns and Notes. A board would have a name, the name of its author and colour, for customisation purposes. Moreover, it would have a list/number of columns. Each column in a board would have a name and a list/number of notes. Lastly, each note would have a title, the note content and colour. A diagram can be seen below:

Database Design Image

Database

As mentioned previously in the brief, we were still considering which database to use for our application. The two choices had been MongoDB and SQL. SQL seemed to fit our design well, and both of us were more comfortable with it, having used it before. Therefore, using our models from above, we created Board, Board_Column and Note tables. Board_Column linked to Board with a BoardID as a foreign key and Note linked to a Board_Column in the same way. We used Board_Column instead of Column as that conflicted with SQL terms. Our table can be seen below.

		CREATE DATABASE ideaBoard;

USE ideaBoard;

CREATE TABLE User
(
	user_id INT NOT NULL AUTO_INCREMENT,
	email VARCHAR(50) NOT NULL,
	UNIQUE(user_id),
	UNIQUE(email),
	PRIMARY KEY(user_id)
);

CREATE TABLE Board
(
	board_id INT NOT NULL AUTO_INCREMENT,
    user INT NOT NULL,
	FOREIGN KEY (user) REFERENCES User(user_id),
	name VARCHAR(100) NOT NULL,
	UNIQUE(board_id),
	PRIMARY KEY(board_id)
);

CREATE TABLE Board_Column
(
	board_column_id INT NOT NULL AUTO_INCREMENT,
    board INT NOT NULL,
	FOREIGN KEY (board) REFERENCES Board(board_id),
	title VARCHAR(100),
	UNIQUE(board_column_id),
	PRIMARY KEY(board_column_id)
);

CREATE TABLE Note
(
	note_id INT NOT NULL AUTO_INCREMENT,
    board_column INT NOT NULL,
	FOREIGN KEY (board_column) REFERENCES Board_Column(board_column_id),
	content TEXT,
	UNIQUE(note_id),
	PRIMARY KEY(note_id)
);
	

Once we started prototyping our application, we discovered that it would be extremely tricky to work with node.js and MySQL. It was certainly possible, but the amount of effort required to make things work was deemed too costly to justify using an SQL database. Instead, a document-based database would be ideal, and this was available in the form of MongoDB. Therefore, we switched to MongoDB. In Mongo, we tried to initially just use one document for each record, meaning a board document, containing its values and a list of column documents which in turn contained note documents. The schema of this can be scene below.

var ideaboardSchema = new Schema(
			{
				id : ObjectID,
				name : String,
				author : String,
				colour : String,
				columns : [
					{
						id : ObjectID,
						name : String
						notes: [
							{
								id : ObjectID,
								title : String,
								content : String,
								colour : String
							}...
						]
					}...	
				]
			}
		);

This seemed to be a good idea but we faced some trouble while retrieving individual notes or columns from a board. One could easily retrieve the whole list but doing that to change to only one document inside the list was considered inefficient and wasteful. Instead, the re-design of our database was similar to the SQL one, with the ids of boards stored in column documents etc. The final schema is as follows.

var boardSchema = new Schema(
				{
					id : ObjectID,
					name : String,
					author : String,
					colour : String	
				}
			);

			var columnSchema = new Schema(
				{
					id :  ObjectID
					name : String,
					board : ObjectID
				}
			);

			var noteSchema = new Schema(
				{
					id : Object ID,
					name : String,
					content : String,
					colour : String,
					column : ObjectID
				}
			);

Design

As mentioned before, the back-end was present to serve the GET , PUT , POST and DELETE requests from our web application. We wanted to be able to display all the boards we had, and then, if needed, a specific board requested with id. This would also apply to the columns and notes belonging to each board. With this in mind, we created a number of routes which can be seen below.

Using /boards will return a list of all the boards contained in the database.
GET boards request
If we enter an id next, /boards/:boardId , this will give us a board with the requested id.
GET specific board request
Moivng on, /boards/:boardId/columns would give us a list of columns, and calling an id on that will give us a specific column.
GET columns request
Lastly, using /boards/:boardId/columns/:columnId/notes and adding /:noteId to it will give us the notes and a specific note within this board column respectively.
GET notes request

Server

Inspiration

The creating of the node server was done with the help of a tutorial which involved creating a wine factory.

Structure

Making a RESTful API also involved making use of different, useful node modules. The two modules used in our application were mongo and express modules. Mongo was obviously used to communicate with our database while Express is a lightweight node.js web application framework, providing the basic HTTP infrastructure to easily to create REST APIs. It allowed us to set which methods would be called by each of the requests that we mentioned in our design. An example of this is assigning a get request to /boards/ to call the method to retrieve a list of all the different boards.

var express = require("express");  //import the module

var app = express();				//instantiate into a variable

//requests for boards
app.get('/boards', boards.findAll);						//GET request
app.get('/boards/:boardId', boards.findById);			//GET request
app.post('/boards/',boards.addBoard);					//POST request
app.put('/boards/:boardId',boards.updateBoard);			//PUT request
app.delete('/boards/:boardId', boards.deleteBoard);		//DELETE request

//columns
app.get('/boards/:boardId/columns', columns.findAll);					//GET request
app.get('/boards/:boardId/columns/:columnId', columns.findById);		//GET request
app.post('/boards/:boardId/columns',columns.addColumn);					//POST request
app.put('/boards/:boardId/columns/:columnId',columns.updateColumn);		//PUT request
app.delete('/boards/:boardId/columns/:columnId', columns.deleteColumn);	//DELETE request


//notes
app.get('/boards/:boardId/columns/:columnId/notes', notes.findAll);					//GET request
app.get('/boards/:boardId/columns/:columnId/notes/:noteId', notes.findById);		//GET request
app.post('/boards/:boardId/columns/:columnId/notes',notes.addNote);					//POST request
app.put('/boards/:boardId/columns/:columnId/notes/:noteId',notes.updateNote);		//PUT request
app.delete('/boards/:boardId/columns/:columnId/notes/:noteId', notes.deleteNote);	//DELETE request

The tutorial had just one table of wines in their collection, but we had three tables. Therefore, when we created methods for retrieving, editing and deleting database records, and wrote them into separate files to imported in the main server file for modularity and a nicer structure.

boards = require("./routes/boards.js");
	columns = require("./routes/columns.js");
	notes = require("./routes/notes.js");
				

Database operations

Connecting to database

We added a separate file to handle connection with the database. This way, we only connect to our database once when start our server, calling db.openDabase(databaseName) .

var mongo = require("mongodb");

var Server = mongo.Server,
	Db = mongo.Db,
	BSON = mongo.BSONPure;

var server = new Server('localhost',27017, {auto_reconnect: true});

exports.openDatabase = function(dbName){
	db = new Db(dbName,server);
	db.open(function(err, db){
	if(!err){
		console.log("Connected to '" +dbName+ "' database ");
		db.collection('boards',{strict:true}, function(err, collection){
			if(err){
				console.log("The 'boards' collection doesn't exist. Creating it with sample data...");
				populateDB();
			}
		});
	}
});
};

As we can see in our above function, we create dummy records if the database did not exist before.This was a simple, yet elegant function, allowing us to create neat sample data, and test our front-end.

var populateDB = function(){
var boards = [];
	var columns = [];
	var notes = [];
	
	for(var i = 0; i<5;i++){
		var boardId = new BSON.ObjectID();
		boards.push({
			_id: boardId,
			name:"testBoard"+i,
			author:"testAuthor"+i,
			colour:"testColor"+i
		});
		var tempColumns = [];
		for(var j = 0;j < 5; j++){
			var columnId = new BSON.ObjectID();
			columns.push(
			{
				_id: columnId,
				board:boardId,
				name: "testBoard"+i+"testColumn"+j
			});
			for(var k = 0;k < 5;k++){
				var noteId = new BSON.ObjectID();
				notes.push(
				{
					_id: noteId,
					column: columnId,
					name: "testBoard"+i+"testColumn"+j+"testNote"+k,
					contents: "This is a test note",
					colour: "testColour"+k
				});
			}
		}
	}

	db.collection('boards', function(err, collection){
		collection.insert(boards,{safe:true}, function(err, result){});
	});
	db.collection('columns', function(err, collection){
		collection.insert(columns,{safe:true}, function(err, result){});
	});
	db.collection('notes', function(err, collection){
		collection.insert(notes,{safe:true}, function(err, result){});
	});
};
Retrieving records

For each type of record (board, column, note), we implemented methods to retrieve specific records or all of them. For the board database, we used standard methods using the tutorial previously mentioned.

exports.findById = function(req, res){
	var id = req.params.boardId;
	console.log("Retrieving board : " + id);
	db.collection('boards',function(err, collection){
		collection.findOne({'_id':new BSON.ObjectID(id)}, function(err, item){
			if(err){
				console.log(err);
			}else{
				res.send(item);
			}
		});
	});
};

exports.findAll = function(req, res){
	db.collection('boards', function(err, collection){
		collection.find().toArray(function(err,items){
			res.send(items);
		});
	});
};

However, for notes and columns, we had to find the columns relating to the board and note relating to the column. We have shown the methods for columns below.

exports.findById = function(req,res){
	var id = req.params.columnId;
	console.log("Retrieving column : " + id);
	db.collection('columns',function(err, collection){
		collection.findOne({'_id': new BSON.ObjectID(id)}, function(err, item){
			if(err){
				console.log(err);
			}else{
				res.send(item);
			}
		});
	});
};

exports.findAll = function(req,res){
	var boardId = req.params.boardId;
	console.log("Columns for board with id: " + boardId);
	db.collection('columns', function(err, collection){
		collection.find({'board': new BSON.ObjectID(boardId)}).toArray(function(err,items){
			res.send(items);
		});
	});
};
Adding new records

Again, it's simple to add a new board. We retrieve the board json from the body of the request and add it.

exports.addBoard = function(req,res){
	var board = req.body;
	console.log('Adding board: ' + JSON.stringify(board));
	db.collection('boards', function(err, collection){
		collection.insert(board, {safe:true},function(err,result){
			if(err){
				res.send({"error":"An error occurred - " + err});
			}else{
				console.log("Success : " + JSON.stringify(result[0]));
				res.send(result[0]);
			}
		});
	});
};

When adding notes and columns, we add the column and board id on client side respectively. However, that did not work properly, so we need to convert the foreign id into an Object ID before adding it to the database. Otherwise, these records would not be retrieved properly.

exports.addColumn = function(req, res){
	var column = req.body;
	console.log('Adding column:' + JSON.stringify(column));
	column.board =  new BSON.ObjectID(column.board);		//converting to ObjectID
	db.collection('columns', function(err,collection){
		collection.insert(column, {safe:true}, function(err, results){
			if(err){
				res.send({"err":"An error occurred - " + err});
			}else{
				console.log("Success : " + JSON.stringify(results[0]));
				res.send(results[0]);
			}
		});
	});
};
Updating records

When updating records, we would need to delete the id from the json object we would received from the client. Not doing so would cause different errors. This was done each of the different tables. The other difference between boards and columns/notes was converting foreign key ids to Object Ids, which has been shown above.

exports.updateBoard = function(req,res){
	var id = req.params.boardId;
	var board = req.body;
	delete board._id;
	console.log("Updating board: " + id);
	console.log(JSON.stringify(board));
	db.collection('boards', function(err, collection){
		collection.update({"_id":new BSON.ObjectID(id)}, board, {safe:true}, function(err, result){
			if(err){
				console.log("Error updating board - " +err);
				res.send({"error": "An error occurred - " + err});
			}else{
				console.log("" + result + "document(s) updated");
				res.send(board);
			}
		});
	});
};
Deleting records

Deleting records for each table is similar. We just need to retrieve the id from the body and delete the record. At this moment, we did not implement functionality of deleting all the columns related to a board if it were deleted, nor the same for notes for a column

exports.deleteNote = function(req, res){
	var note = req.body;
	var id = req.params.noteId;
	console.log("Deleting note : " + id);
	db.collection('notes',  function(err, collection){
		collection.remove({"_id":new BSON.ObjectID(id)}, {safe:true}, function(err, result){
			if(err){
				res.send({"error":"An error has occurred - " + err});
			}else{
				console.log(result + " document(s) removed");
				res.send(note);
			}
		});
	});
};

Testing

To an extent, we tested our server using cURL , follwing instructions from the tutorial. Some example tests can be seen below

  • Get all boards: curl- i -X GET htp://localhost:3000/boards
  • Get board with _id value of 5478fec3e929b4f0311c4a9b: curl -i -X GET http://localhost:3000/boards/5478fec3e929b4f0311c4a9b
  • Delete board with _id value of 5478fec3e929b4f0311c4a9b: curl -i -X DELETE http://localhost:3000/boards/5478fec3e929b4f0311c4a9b
  • Add a new board: curl -i -X POST -H 'Content-Type: application/json' -d '{"name": "New Board", "author": "New Author"}' http://localhost:3000/boards
  • Modify board with _id value of 5486e09273ac4cbc289c1124: curl -i -X PUT -H 'Content-Type: application/json' -d '{"name": "Board name new", "year": "Author change"}' http://localhost:3000/boards/5486e09273ac4cbc289c1124

    Front-end

    User Interface

    IdeaBoard uses a natural user interface; an interface which is initially invisible, but whilst the user explores the application they discover and learn the supported functions. As such, upon first seeing the board the user sees only a series of notes but when hover over the notes with a mouse they can see UI interface elements to change colour and delete notes. When the user selects text a menu appears under their cursor allowing them to:

    • Create bullet points and numbered lists
    • Make text bold, underline or italic
    • Create heading
    • Make links

    Upon hovering over the end of a list or column a user discovers the new note or new column button. All of these appearing interface elements are animated to fade in and out, this is so that the user is not forced into sudden disorientating state changes when editing a board.

    Whilst we are very pleased of our user interface it is heavily reliant on mouse cursors in order to operate. Thus it is not suitable for mobile use. Our plan in the future is to implement a mobile application in native code the utilises the same API as our web application. When a mobile user accesses the site they can then be redirected to the relevant app store to download a mobile version.

    Despite not implementing a mobile version of the site we have implemented a responsive design to be flexible to different display sizes. Our responsive design was implemented using the Bootstrap framework. Below is an example of the board selection screen responding to width changes.

    The board selection screen on a narrow screen
    Board selection screen on a wide screen

    Front-end libraries

    One the skill we aimed to improve is the use of JavaScript libraries for single page applications. With extensive use of libraries for user interface elements we were able to easily implement attractive controls far more easily than if we had coded them ourselves.

    In addition to the libraries used below this page uses Prismjs to syntax highlight code examples and Tocify to create a table of content.

    Backbone.js

    Where?

    To create the in app router allowing history and navigation on a single page web application

    Why?

    We wanted to use a lightweight MVC web framework to give structure to our application and Backbone is very lightweight and we had previous experience with it. However, we found using a MVC framework for our site too cumbersome and simply developed it without with the exception of the in app router which we needed for basic functionality.

    Example
    
    var AppRouter = Backbone.Router.extend(
    {
    	routes:
    	{
    		""			: "listBoards",
    		"boards/new":"newBoard",
    		"boards/:id":"showBoard",
    		"*actions": "defaultRoute"
    	}
    });
    
    var app_router = new AppRouter();
    app_router.on('route:showBoard',function(id)
    {
    	showBoard(id);
    });
    
    
    The code in main.js used create a router, define its routes and follow the showBoard route.

    jQuery

    Where?

    Used throughout the application for DOM manipulation, ajax requests and smoothing over API differences between web browsers.

    Why?
    • Both our team members had extensive experience with jQuery
    • It allows for more concise code (it's marketed as'The Write Less, Do More, JavaScript Library.')
    • Its the most commonly used front-end we library so its easy to find help
    • Other libraries(jQuery notebook and JQuery color picker) we wanted to use were jQuery extensions
    Example
    
    <!-- DOM manipulation - Creating the divs that hold the notes -->
    colContainer.append("<div class=\"notesContainer\">");
    
    <!-- An Ajax call to get the board name and author-->
    $.get(apiURL+'/boards/'+boardID,function(data)
    {
    	$(".userField").text(data.author);
    	$(".boardField").text(data.name);
    });
    
    

    JQuery notebook

    Where?

    So that the notes support rich text, when the user highlights a note a bubble allowing editing action applied to the selected text.


    Why?

    Used because the editor for jQeuery notebook only appears when text is selected it fulfilled our goal of creating a natural user interface. Also it allows all the rich text function we wish to use.

    Example
    
    $(textArea).notebook();
    
    $(textArea).on('contentChange', function(e)
    {
    	var textArea = e.originalEvent.target;
    	var note = $(textArea).parent();
    	var id = note.attr('data-noteID');
    	var colID = $(note).parents('.boardColumn').attr('data-columnid');
    	updateNote(id,colID,currentBoardID);
    });
    
    
    Code used to initialise jQuery Notebook and attach an event listener to update on text changes
    Problems

    We encountered incompatibilities between jQuery notebook and Twitter bootstrap with regards to the position of the pop up bubble. Upon discovering this the site already had already been developed using both these libraries. We found however, that these errors only occurred when the parent elements of the notebook used relative positioning so it was relatively simple to work around.

    jQuery notebook with relatively positioned columns causing the bubble appear up and to the right of selected text (during development)

    jQuery Color Picker

    Where?

    Used to edit the colour of the notes, was slightly modified to support our selection of colours and add context to the callback function


    Why?

    We chose this because it supports only a limited number of colours rather than a any shade. This means that the UI is simpler and the user can only pick colours that complement other colours and work with the black text.

    Example
    
    $(colorPicker).empty().addColorPicker(
    {
    		clickCallback: function(c,picker)
    		{
    			$(picker).parent().css('background-color',c);
    			updateNoteDiv(picker.parent());
    		}
    });
    
    
    Code used to initialise colour picker and provide it with a callback to update the note and server on modification

    Twitter Bootstrap

    Used for page Layout and design. The top navigation bar is implemented using bootstrap components. We also used some of the bootstrap glypicons for UI elements, because these are vector icons our site supports high-DPI displays. Bootstrap was also used in the project brief and manuals for the site.

    Example
    
    boardList.append('<h1 class="text-center">Available Boards<br>');
    $.get(apiURL+"/boards",function(data)
    {
    	var boards = '<ul class="list-inline">';
    	$.each(data,function(key,value)
    	{
    		boards+= "<li class='boardLink' data-boardID='"+value._id+"' > < a class='board plain thumbnail text-center' href='#boards/"+value._id+"' >  <h3> "+value.name+" < br > < small > "+value.author+" < small > < /h3 > < /a > < /li >";
    	});
    	boards+=" < /ul> ";
    	boardList.append(boards);
    });
    	
    
    
    Code used to display boards generating markup with the bootstrap classes text-center and list-inline.

    Testing

    In order to provide test data on start up the server checks to see if a database exists, and if it does not it generates a set of test data automatically.

    We chose to target support for the previous two versions of the four most popular desktop browsers.Because our user interface is not suitable for touch screens we chose not to support mobile browsers.

    • Internet Explorer 9+
    • Google Chrome 37+
    • Firefox 30
    • Safari 6.2

    We developed the site in Google Chrome because we were most productive with their development tools.

    Skills Tables

    We have included the table of our skills from the project brief.

    The skills we wanted to improve are listed below, in decreasing priority.

    1. Creating a responsive user interface with HTML5, CSS3 & JavaScript
    2. Web patterns and frameworks such as model-view-controller
    3. JavaScript libraries/frameworks for single page applications
    Skills table before the project
    Web Development Skills None Beginner Intermediate Expert Guru
    Creating a responsive user interface with HTML5, CSS3 & JavaScript
    Use of an HTML template engine and standard elements such as web controls or widgets
    Role-based security and logging for improved management of information security
    Declarative validation of user input
    Asynchronous web server communication and updates
    XML or JSON formatted data import & export
    Rich content, embedded audio and video, interactive graphics
    JavaScript libraries / frameworks for single page applications
    Object-relational mapping tools to simplify database coding
    Web patterns and frameworks such as model-view-controller
    Automated tools for testing accessibility and portability
    Database transactions for isolation and consistency
    Performance management and scalability

    Out of the targeted skills to improve, we definitely managed to improve creating responsive user interface. Micheal mostly worked on this and took his level from an Expert to Guru. We were not able to do a lot of work in MVC, but we used routing from Backbone.js to improve our skills on creating single page applications from beginner to intermediate. Apart from that, we did see an increase in skills from working on the back-end server, with both improving from beginner from none to intermediate. This also applies to JSON formatted data import and export, again improving from beginner to intermediate.

    Skills table after the project
    Web Development Skills None Beginner Intermediate Expert Guru
    Creating a responsive user interface with HTML5, CSS3 & JavaScript
    Use of an HTML template engine and standard elements such as web controls or widgets
    Role-based security and logging for improved management of information security
    Declarative validation of user input
    Asynchronous web server communication and updates
    XML or JSON formatted data import & export
    Rich content, embedded audio and video, interactive graphics
    JavaScript libraries / frameworks for single page applications
    Object-relational mapping tools to simplify database coding
    Web patterns and frameworks such as model-view-controller
    Automated tools for testing accessibility and portability
    Database transactions for isolation and consistency
    Performance management and scalability

    Mohammad Ali Khan (mak1g11)
    Michael Terry (mrt1g11)
    Both