Managing Coordinate Systems

Coordinate systems is maybe not something you are faced with every day, and might lead you more to the gaming industry, but nevertheless are coordinate systems something you will face sooner than you think. The pixels of your screen for instance already define a coordinate system. This system has the zero point in the left top corner which is similar to how we read in standard languages such as English. We start at the top and go to the bottom, and on each line we read from left to right.

The coordinate system of your screen is a bit special, as it does not allow for negative values and works only with natural numbers. Because of this implicit coordinate system something as simple as showing something on the screen already requires a conversion or mapping. In this blog post I will go into more detail as to what problems can arise and what you should take into account.

Continue reading

Advertisements

Router Application

Recently we had the problem that some application that was running at the customer was not able to talk to multiple instances of our application. There were a lot of reasons why we wanted a separate instance of our application, even though we could have combined them into one. The main reasons was the actual physical differences between the things they had to manage. But changing the software of the customer would be too costly, so instead we investigated whether it would be possible to introduce a kind of router to handle this.

Continue reading

The Hard Part Of Automation: Manual Interaction

As more and more things are being automated, our daily life becomes easier. In many cases however we have a mix of automation and manual actions, or we want to be able to intervene or correct when automation goes wrong. Whatever reason, a fully automated system is often still not feasible, and thus manual interactions needs to be allowed and taken into account during design and implementation. But what exactly are the implications of this?

Continue reading

Software Fixes For Hardware Problems

Dealing with a physical system is pretty challenging as you have to take into account the hardware limitations. There is a big misconception that software is only limited by your imagination, and that whatever you think of you can just write it. But when you are dealing with software that completely deals with digital data and has no presence in the physical world, you do have a lot more freedom compared to when you have to take the physical world into account.

Besides the limitations you also have to take into account noise, inaccurate values and even hardware failures. You will often need to write your software such that it can deal with all of these problems, even though these are not caused by the software itself, so we already see a certain level of ‘fixing’ hardware problems. Because if the hardware would be accurate and immune to failure, your software could just rely on the values it gets. But since this level of correctness is impossible in the real world, you have to fix it in the ‘virtual’ world.

Continue reading

Building A Hadoop Cluster With Docker – Part 3

This part continues where we left of with part 2, we will examine the other ways on how to upload files to a Hadoop cluster:

  1. Using the command line.
  2. From Java using the DFSClient
  3. From Java using the FileSystem

If you did not read the previous part, I highly recommend you to do so before continuing. Before diving into the new work, lets summarise what we learned from the previous part.

  • We had to fix the hostname of the container.
  • The mapping of ports caused a problem.
  • The filesystem permissions are very restrictive to non-root users.

The first two points means that the cluster can not be used as it was designed in part 1, and is restricted to a single datanode.

Continue reading

Building A Hadoop Cluster With Docker – Part 2

After setting up a HDFS cluster in part 1, it is now time to put some data in the cluster. There exists multiple ways to upload files to a Hadoop cluster:

  1. Using the Web UI of the namenode
  2. Using the REST API.
  3. Using the command line.
  4. From Java using the DFSClient
  5. From Java using the FileSystem

Treating them all in a single post would make the post too long (in my opinion). Therefor I have chosen to split it up into two separate posts, meaning that in this blog post I will only discuss the Web UI and REST API (which should be nearly identical). The other approaches will be handled in the next.

Even though you may not be interested in using the Web UI or the REST API to upload files to you Hadoop cluster, I recommend reading through this post as problems encountered here, will come back in the next one. Even more important though are the fixes and changes that are applied to get it working as they will have an impact on the other ways of uploading files as well.

Continue reading