Over the past few months whilst development has been on-going with the Moodster API, Glen has been prototyping various JS Frameworks for the frontend of the application. The requirements for this were a little more complex than the API.

We wanted the frontend to be able to have ideally one codebase that would be cross platform for  mobile, web and possibly desktop too.

Glen wanted to give some different frameworks a try so he could find the best fit but also to gain some experience in various frameworks as he was looking for a new job at the time.

Some of the options that he's been trying out are:

• Quasar - This is a framework built on top of vueJS with great support for deploying to multiple targets from one codebase.
• ReactNative - I don't think this is likely to be chosen as it's not very easy to have on codebase for multiple platforms.
• Angular - This is another viable option which also supports Native apps on mobile. Larger popularity than Quasar at the moment as well.

Not sure which way it's going to go at the moment but we need to get this nailed down soon so we can start making progress.

I've completed it!

I have done the 4 required PR's for Hacktoberfest, which means I am eligible for the t-shirt. It actually wasn't as easy as I thought. I could probably have been a bit naughty and broken down my PR's into much smaller pieces of work to try and bump the numbers up, but I would have just been cheating myself out of doing as much work as I have.

So to give a summary of the new functionality I've added to Moodster API over this Hacktoberfest, here are my PR's:

​October 02, 2019 00:44 Added the moods table, including default data.
October 02, 2019 21:31 Added test to cover none type token payloads to
October 18, 2019 18:14 Added team creation deafault admin user
October 19, 2019 16:08 Member mood addition

​So that's it, there has been some great additions there which push the functionality further than what the previous Flask-Restplus verison had. Furthemore I believe that the API is probably now MVP complete. Looking back at the original blog post that laid out the MVP the API has endpoints which would allow the frontend to perform all of those actions.

Next up is to try and push development a bit more on the web/mobile app front to try and get that to MVP.

That's right Digital Oceans Hacktoberfest is just about to kick off!

I'm going to give it a go and use it to motivate me to do some more work on the moodster API.

​I've made some real progress and the FastAPI version of the Moodster API is almost functionality equivalent to the previous version.

Wish me luck!

A while ago I began trying to dockerise the moodster API application and it was quite difficult to get started as the project was already under way and I wasn't that experienced with Docker at the time.

Since I'm re-writing the Moodster API I decided I should start off with dockerisation and bake it straight into my development process.

Fortunately the FastAPI documentation already comes with pre-built examples and docker containers for running FastAPI in Docker.

One of the main reasons I've been wanting to do this is that I've recently been looking into the 12 factor app. The 12 factor app is essentially a set of guidelines for how to develop applications that are to be ran as services.

The guidelines in place are to allow you to create services which are as maintainable and scalable as possible. The rule in particular that has driven me to do dockerisation of moodster is called Dev/Prod Parity. This essentially says that the development, staging and production environments should be as similar as possible.

In the Flask-Restplus version of the Moodster API I was using sqlite for development and then I was using PostgresSQL for the unit tests and eventually for deployment. This caused me issues from the off where I was hitting issues with differences between how Postgres and sqlite worked with SQLAlchemy and database migrations. It also made the configuration more confusing as I was having to override lots of things depending on dev vs testing. This is not ideal!

The dockerisation has gone fairly smoothly and I now use docker in development. This allows me to just use Postgres for all the environments and it's just another thing that I now don't have to worry about.

I did find that it slowed some development speed initially but now that I'm used to the docker commands a lot more it's no longer a barrier. Also the amount of time it is going to save me when it comes to deployment means it will be well worth it.

​Cheers

This is not something I normally do and to be quite honest in a work environment I wouldn't do this, but I've decided to re-create Moodster using FastAPI.

FastAPI is a small API focused web framework built on top of a few rather large projects:

• Starlette - An ASGI Web Framework
• Pydantic - Data validation using Type hints

There are a few reasons I've done this:

First of all Flask-Restplus was already giving me issues with data validation as mentioned in this post. They didn't seem to have a proper answer for the correct way to do this yet. On top of that I saw how clean it looked with pydantic to use type hints for data validation.

I have also been hearing about many different web frameworks that are now built with asyncio out of the box and thought it would be a good idea to future proof a little by choosing a framework that was capable of this. Realistically I'm not even going to use the async capabilities straight away as I don't think it's necessary for the scope of the project. It is certainly a plus knowing that it is there if I need to boost performance in that way.

Finally back in this blog post I said one of the reasons I didn't opt for a web framework that was ASGI was due to the lack of the ORM libraries not being async capable. Well the project that the starlette web framework is under also has libraries for async capable ORM's. This final piece put me at ease enough to make the switch.

There is a lot of work ahead essentially re-inventing functionality that already exists but it's also going to give me a chance to do things in a slightly better way as I've learnt quite a bit since starting out the Moodster API project.

A while ago I posted a blog post about a hackathon project I did for work. In this project it required me to run multiple SSH sessions to many machines at once to query the state of the system.

Recently I've been listening to a lot of the Talk Python To Me Podcast, and hearing all about the new async features of Python 3. This got me thinking about this project and how this would be the perfect candidate for some async functionality. Why is this?

So we had a fairly large number of machines that we were wanting to run these SSH Commands on and many of those machines were simply not turned on or currently unavailable for SSH connections. This caused an issue as we simply had to wait for the timeout to occur which was down to about 1 second to ensure we gave enough variance for slow network.

During this 1 second window each of the worked threads were simply waiting there counting down the milliseconds. This was extremely inefficient!

So Python introduced the async and await keywords but unfortunately it is likely going to be a while before there are many ORM's that are async capable. This is really handy as I had decided that the API was going to be fairly stateless other than the in memory cache, which means implementing this new approach should be fairly simple.

So I used Flask for the web framework for the API and there is now a drop in replacement being developed called Quart. This enables the async and await functionality into that same Flask development style that everyone loves with Flask.

​Along with Quart is another python package called asyncSSH which allows the use of async and await for SSH Connections.

So the actual changes required to test out this new approach hopefully shouldn't be too much effort.

The big question is will this actually improve performance?

At work we occasionally have a Hackathon day where everyone in the lab is able to work on any project they like whether that's just practising some Test Driven Development(TDD) or creating something that could help with your day to day work.

So for this Hackathon I got together with Adrian Moldovan and Tom Fletcher to work on a project that would hopefully address a major issue we have at work.

Our problem is that our product has many different platforms and we don't have all the different platforms in each of the teams so we often have to share our hardware. The problem with this is that in juggling the hardware between teams some hardware can often get lost along the way.

The major problems we have are:

• Finding the a specific platform that you need to test.
• Finding hardware on a specific code level.
• Finding hardware that is in a good state for you to borrow.
• Figuring out which hardware isn't being utilised.
• Finding out who you should talk to about borrowing a piece of hardware.

As you can see we have quite a variety of kit on many different levels and this information is invaluable for quickly finding the piece of hardware you require. 

So let's talk about the tech that has been used for this project. Firstly I've been reading a lot about microservices and was intrigued by this concept of splitting out what would normally be on large application into smaller portions. By no means is this a microservice based architecture for this project as it lacks some of the required infrastructure to be that way.

This project is split into two smaller pieces:

1. A Python REST API which performs the same ssh command to multiple machines at once and combines the result into a single response.
2. A React Web Frontend which displays a table of information for multiple machines.

So the basic idea for the project was to run many of the same ssh command to all of the machines and get a point in time capture of all of the state required to solve the above problems. To achieve this we decided to use threads to be able to kick off multiple ssh sessions at once.

I tend to avoid using multi-threading whenever I can because it massively increases complexity but in this situation it seemed like the correct thing to do.

So I really liked the idea of the REST API being a really simple worked that simply ran commands on multiple machines and output the result.

I decided that we should try and keep this fairly stateless and so the only real state that the API has is a simple in memory cache which wouldn't even be necessary if it wasn't for the sheer number of machines that we are trying to perform the commands on.

The caching itself is only held for 15 minutes for each machine to ensure the data isn't too stale.

So this portion of the project is where all the shiny things happen, without this the user would need to be able to easily read and interpret JSON data which is not a skill many people have.

For ease of use we added a search functionality along with some filter buttons which will allow the user to quickly narrow down the choice that they are after. There is also a refresh functionality which will call to the API and tell it to do a full refresh not including cached data. This is mostly for people who really need the most up to date information.

For this project we really wanted to try out using Docker as we wanted to be able to deploy these two applications as simply as possible. After all this was a Hackathon and so it is a perfect opportunity to try and harness new skills.

Both projects were there own separate containers and the frontend simply needed an argument on run which passed in the address for the API.

I think the biggest things I learnt were to do with application architecture specifically how this style of application level separation of concerns allows for extendable and reusable functionality. The API itself can be used by any frontend client. If React is no longer the flavour of the month then a new frontend could be made, or if someone wanted this functionality on mobile then they could still simply call the same API.

This whole idea of containerisation I now see if incredibly powerful, it is much easier developing on the same platform that you are deploying to and also this ability to deploy anywhere that has just the one dependency of being able to run Docker is so useful. 

I will definitely be doing a lot more with Docker in future.

There are always improvements that can be made, specific ones I would like to look into are:

• Support for the OpenAPI /Swagger API Specification.
• Adding Docker Compose support for deploying both of the Containers together.

At work I recently helped organise a Hackathon for the Manchester IBM Lab. I wanted to take some time and do a retrospective of the projects that my team worked on. Our team consisted of Toby Fleming, Manuel Cantu Reinhard and myself. We ended up doing two projects, one which was more of an electronics project and the other which was a software development project. 

We have a doughnut rota at our office that means every Friday whoever is down for that week has to go and buy some Krispy-Kreme doughnuts for the lab. The steps to do this are:

1. Go and buy the Doughnuts.
2. Bring the Doughnuts to the kitchen.
3. Send out an e-mail to let everyone know that they have arrived.

We thought we could do better than this so we came up with DaaS!

We first came up with the idea that when any sugary treat is delivered to the kitchen there should be a way to instantly let everyone know that they have arrived. Nobody wants to be waiting an extra ten minutes for someone to get to their desk and compose an e-mail to be informed of doughnut arrival.

We set off to work hooking up the raspberry pi with the flick switch and "Big red button" that we had ordered. At various points through the day we stopped to discuss design decisions and any problems with a whiteboard session.

After hooking up the electronics and doing some simple tests we started working on the implementation. We used a simple python GPIO library for interfacing with the flick switch and button.

So here is what it looks like fully working with the switch activated(obviously the Pi goes inside the box). I'm actually really proud of this as it was a pretty cool project to work on.

Thanks.

I am going to give a small example about how to get session timeout to work for Flask while using the Flask-Login extension.

I have read a few stack overflow posts that show how to do this in pieces but I wanted to summarise this into one example.

Here is the @app.before_request function that will allow you to have session timeout

import datetime
import flask
import flask_login


@app.before_request
def before_request():
    flask.session.permanent = True
    app.permanent_session_lifetime = datetime.timedelta(minutes=20)
    flask.session.modified = True
    flask.g.user = flask_login.current_user

The flask.session.premanent flag and the app.permanent_session_lifetime allow Flask to know that you want the session to expire. If left with only these two then the session will expire every 20 minutes regardless of whether the user has been active. Realistically you would want the session to expire after 20 minutes of inactivity, which is what the flask.session.modified flag is for. Each time there is a request the flag gets set to True which effectively resets the session timeout timer. The final line retrieves the logged in user from flask_login and sets the Flask global user so that it can be used by the Jinja templates.

As I said that this was to be used with the Flask-Login extension I wanted to point out something that could potentially catch you out. Flask-Login has a "remember me" functionality that is set at login time, the use of this functionality can mess up the session timeout and make it appear as though it does not work. To avoid this you need to look at the login code and ensure that the remember flag is not set to True:

flask_login.login_user(user, remember=False)

If you want to use the "remember me" functionality then you may need to look into ensuring that the remember me cookie duration is changed but that is out of the scope of this example.

Hope that helps,
​Thanks

Okay so it's been a while since my last post and I was going to say that the reasons why I've not been doing anything is because I've been busy at work. While this is true I thought I would take this blog post to be more constructive and delve into what I've been learning/developing whilst at work.

At work I have been the lead developer for a small web application that is going to be used internally to help customer support. I have been responsible for creating the design document, discussing with other developers/stakeholders to develop the requirements and iterate over the design to come to a final solution.

In the early stages of the project I was a sole developer working on the application, I chose Flask (a Python web micro-framework) as the tool for the job. Flask is a great tool for creating web applications, as it takes a lot of the complexities away whilst not being as all encompassing as Django.

As Flask is a micro-framework it has a vast number of extensions that allow you to add extra pieces of useful functionality into your web application. Here are a list of some of the extensions I have used:

• Flask-SQLAlchemy (provides better integration of SQLAlchemy (a Python DB Abstraction Layer))
• Flask-Login (provides user based logins and also deals with session management)
• Flask-Uploads (provides easier file upload functionality)

As the project started to take shape I was given some extra development resources that I would need to lead and give direction to so the project could progress. 

Looking back I didn't do very well as a team leader at the very beginning of the project. I had various problems early on which I now know ways in which to deal with them:

Initially I didn't give enough direction, forgetting that they hadn't been involved in the early development and didn't understand the reasons that some of the design decisions were made.

This lack of direction was made much worse because I was mainly communicating via e-mail or Instant messaging. This ended up with a lot of back and forth and a lot of frustration on both sides.

​After getting some guidance from my manager I decided to set up regular conference calls to actually talk to my teammates in Pune and this ironed out a lot of the issues and also freed up time to actually get some development done.

I decided to use Python for the development as there was a lot of other Python developers in the lab which would allow me to seek guidance when I needed it. I didn't take into account that other developers would be working on the project and that they might not have Python experience.

​This lack of experience from my teammates was rather difficult to overcome and there were a lot of large code reviews early on explaining exactly why we do certain things, like string formatting for example:

print "This is my string with a variable: {}".format(my_variable)

print "This is my legacy string with a variable: %d" % my_variable

These small differences along with issues with different people using different tab lengths and not using space-tabs caused the reviews to grow a lot.

After doing a few of these code reviews and also talking to my team via the regular conference calls these issues soon ironed out. I also made it mandatory that all my team must run pylint code analysis before submitting for review. In future I will ensure that all my fellow developers have a style guideline or code analysis application like pylint from the start.

As I was having to do so many large code reviews early on it meant that these would often take me a while to do. I started off doing these code reviews in the morning, as Pune are five and a half hours ahead of us this meant that they were already half way through their next day when I would get to doing the code reviews. 

I started doing code reviews as soon as the team had them for me, this would mean I could get it reviewed and comments back to them to address in their morning before I got to work. I would then use evenings whilst they were offline to do my own development work.

​This was only a small change but it made the team much more fluid and allowed us to get more tasks done sooner.

So all in all I've learnt a lot, I've now got more skills in Python, Flask and various other frameworks. What I think is more important is that I've learnt how to lead a team much better and I think any future projects would go much more smoothly now that I've learnt these lessons, even if I had to learn them the hard way. I still think I am far from a great leader but I'm a lot closer than I was before this project.

​Cheers :)

I just wanted to put a quick post up about some prototyping I am doing at work. I have been given the task of creating a dummy web application which has taken me into a realm I am not familiar with at all so I thought I would talk about it a little. 

I basically had total control over what technology I used for this application and someone suggested to use Perl as people had used it for other application we have. I really didn't want to use Perl and there is a couple of reasons for this:

1. If I have to learn a new language I would rather it be one that I like the look of
2. I would rather learn one that is becoming more popular rather than one that is dying off.
3. There are a few people in work that are willing to help with this little prototype and they have more expertise with python.
4. Python has SQLAlchemy which supports both database types we will need to use.

I will now talk a little bit about the other choices I have made in terms of tech: 
I've decided to use Flask as it is a lightweight web framework which will allow me to do exactly what I need. This web application isn't complex so I didn't really want something large and all encompassing.

As the Web application has to use DB's I've decided to use SQLAlchemy as it seems fairly popular, has support for MySQL(The Test DB I am using for this) but also support for DB2(What we will likely deploy it on). 

Finally virtualenv. This is something I've never used before but it seems like it is going to be really useful as deployment to other machines can be done without worry as It won't matter about them having the correct python version or flask version installed. 

All in all it's been great, I've had to learn a lot so far and I'm finally getting the hang of how it all fits together. I've been lucky enough to have another application to use for guidance through this development and so I should have it up and running in no time.