Category: Agile

Node Info Services - Python Async IO and AsyncSSH

13/6/2019

A while ago I posted a blog post about a hackathon project I did for work. In this project it required me to run multiple SSH sessions to many machines at once to query the state of the system.

Recently I've been listening to a lot of the Talk Python To Me Podcast, and hearing all about the new async features of Python 3. This got me thinking about this project and how this would be the perfect candidate for some async functionality. Why is this?

Timeouts

So we had a fairly large number of machines that we were wanting to run these SSH Commands on and many of those machines were simply not turned on or currently unavailable for SSH connections. This caused an issue as we simply had to wait for the timeout to occur which was down to about 1 second to ensure we gave enough variance for slow network.

During this 1 second window each of the worked threads were simply waiting there counting down the milliseconds. This was extremely inefficient!

Stateless (Hey it worked out)

So Python introduced the async and await keywords but unfortunately it is likely going to be a while before there are many ORM's that are async capable. This is really handy as I had decided that the API was going to be fairly stateless other than the in memory cache, which means implementing this new approach should be fairly simple.

Approach

So I used Flask for the web framework for the API and there is now a drop in replacement being developed called Quart. This enables the async and await functionality into that same Flask development style that everyone loves with Flask.

Along with Quart is another python package called asyncSSH which allows the use of async and await for SSH Connections.

So the actual changes required to test out this new approach hopefully shouldn't be too much effort.

The big question is will this actually improve performance?

0 Comments

Hackathon: Node Information Service

24/5/2019

0 Comments

At work we occasionally have a Hackathon day where everyone in the lab is able to work on any project they like whether that's just practising some Test Driven Development(TDD) or creating something that could help with your day to day work.

The Idea

So for this Hackathon I got together with Adrian Moldovan and Tom Fletcher to work on a project that would hopefully address a major issue we have at work.

Our problem is that our product has many different platforms and we don't have all the different platforms in each of the teams so we often have to share our hardware. The problem with this is that in juggling the hardware between teams some hardware can often get lost along the way.

The major problems we have are:

Finding the a specific platform that you need to test.
Finding hardware on a specific code level.
Finding hardware that is in a good state for you to borrow.
Figuring out which hardware isn't being utilised.
Finding out who you should talk to about borrowing a piece of hardware.

Finished Product

As you can see we have quite a variety of kit on many different levels and this information is invaluable for quickly finding the piece of hardware you require.

Tech

So let's talk about the tech that has been used for this project. Firstly I've been reading a lot about microservices and was intrigued by this concept of splitting out what would normally be on large application into smaller portions. By no means is this a microservice based architecture for this project as it lacks some of the required infrastructure to be that way.

This project is split into two smaller pieces:

A Python REST API which performs the same ssh command to multiple machines at once and combines the result into a single response.
A React Web Frontend which displays a table of information for multiple machines.

Python Flask REST API

Multithreading

So the basic idea for the project was to run many of the same ssh command to all of the machines and get a point in time capture of all of the state required to solve the above problems. To achieve this we decided to use threads to be able to kick off multiple ssh sessions at once.

I tend to avoid using multi-threading whenever I can because it massively increases complexity but in this situation it seemed like the correct thing to do.

Stateless-ish

So I really liked the idea of the REST API being a really simple worked that simply ran commands on multiple machines and output the result.

I decided that we should try and keep this fairly stateless and so the only real state that the API has is a simple in memory cache which wouldn't even be necessary if it wasn't for the sheer number of machines that we are trying to perform the commands on.

The caching itself is only held for 15 minutes for each machine to ensure the data isn't too stale.

React Frontend

So this portion of the project is where all the shiny things happen, without this the user would need to be able to easily read and interpret JSON data which is not a skill many people have.

Search and Filter

For ease of use we added a search functionality along with some filter buttons which will allow the user to quickly narrow down the choice that they are after. There is also a refresh functionality which will call to the API and tell it to do a full refresh not including cached data. This is mostly for people who really need the most up to date information.

Infrastructure

Docker

For this project we really wanted to try out using Docker as we wanted to be able to deploy these two applications as simply as possible. After all this was a Hackathon and so it is a perfect opportunity to try and harness new skills.

Both projects were there own separate containers and the frontend simply needed an argument on run which passed in the address for the API.

Lessons Learned

I think the project went really well and everyone we presented it to was really impressed and wanted to see what else we could get out of it.

Architecture

I think the biggest things I learnt were to do with application architecture specifically how this style of application level separation of concerns allows for extendable and reusable functionality. The API itself can be used by any frontend client. If React is no longer the flavour of the month then a new frontend could be made, or if someone wanted this functionality on mobile then they could still simply call the same API.

Containerisation

This whole idea of containerisation I now see if incredibly powerful, it is much easier developing on the same platform that you are deploying to and also this ability to deploy anywhere that has just the one dependency of being able to run Docker is so useful.

I will definitely be doing a lot more with Docker in future.

Improvements

There are always improvements that can be made, specific ones I would like to look into are:

Support for the OpenAPI /Swagger API Specification.
Adding Docker Compose support for deploying both of the Containers together.

0 Comments

Moodster API Tech Decisions

2/4/2019

0 Comments

I've just started the API portion of the Moodster project and I wanted to talk a little about my decisions for the tech stack.

I've decided I'm going to stick with Python as my language of choice as I have become fairly proficient with it and it does allow you to get things up and running pretty quickly.

I was tempted to try out Golang for this as I have heard that it is well suited to api's/services. I decided against that in the end opting for something that I already know. This comes back round to the whole idea of trying to actually get something completed.

Async or not to Async that is the question.

So I've been listening to the Talk Python To Me lately and heard a hell of a lot about the new async capabilities of python 3. I've been looking around and found Quart, which is a async capable version of the Flask API.

I've been really tempted to start off with this but I have now decided against it. At the moment of writing there are only some of the Flask Plugins which are guaranteed to work with Quart and I think picking a framework that is still in its infancy could also end up tripping me back in the long run for the benefit of potentially getting some speed increases.

This seems to scream out the old adage "premature optimisation is the root of all evil".

Secondly from what I can see at the moment there doesn't seem to be a great story on using the SQLAlchemy ORM with async which would be the main usecase/potential speed increase for asycn and await.

Flask Restplus

I've ended up settling on Flask and the Flask-Restplus plugin as my starting point for this project. Flask Restplus gives you some added functionality for pretty much free when concerned with REST API's.

Some of these are:

Built in OpenAPI Spec Documentation generation.
Response Marshalling.
Request Parsing.

That's my overall starting point with this project, who knows what additional plugins/tools I will venture towards later on.

0 Comments

Moodster - Getting Started

4/3/2019

0 Comments

So I recently posted about an idea I had to create a digital mood marble application which would allow scrums/teams/squads to track their mood over time. I've now come to the point where I want to kick start this project.

Firs things first I need to come up with an MVP (Minimum Viable Product) which I can use to ensure that I actually get to deployment of version 1 without feature creeping.

Team Leader:

Create a Team.
Add a team member to a team.

Team Member:

Join a team.
Submit a Mood for the day.

Anyone in the Team:

View Team Mood Overview/Statistics.

So that is the bare minimum application that I need to create for me to be happy to release it to the public.

0 Comments

Mood Marble Idea

13/2/2019

1 Comment

For a while now I've been thinking about a potential project that I could undertake that is small enough for me to carry all the way to completion/deployment.

This is something has been a struggle in the past as I've tended to choose projects that are simply too large for me to keep the motivation when working on it in my spare time.

Recently I've come up with an idea that I think is small enough in scope to be accomplishable whilst still being a useful project to undertake.

So what are mood marbles?

At work we've been focusing on using Agile methodologies for quite a while and one of the tools that people often use at work is 'mood marbles'. The concept is that each member of the team can place one marble into a jar and the colour of the marble they choose depicts the mood they felt that day.

The whole idea of this, is that you can use this semi-anonymous jar to depict the mood of the team over time. When you come to look at the jar for a current period and the marbles in it are mostly red/orange then you know the team has not been happy recently and it can then open doors to address why that might be.

The problem that I see with mood marbles is:

It doesn't help well for remote team members.
It's not fully anonymous as someone could see you place your marble in the jar.
It doesn't provide any useful metrics for the periods between the jar being emptied.

So this is when I came up with the idea of digitalising this!

Basic Concept

After doing some research I realised that this isn't actually a new project and it's been done before by other companies.

What hasn't been done is creating this mood marble software and making it completely free and open source, so that is what I'm going to do.

The basic idea I have in my head right now is to create a small REST API that will be used to perform all the core business logic/storing of the data, then having thin mobile/web/desktop clients that will simply interact with the API to perform the necessary tasks.

I'm going to really give this a go so watch this space!

1 Comment

Clean Code Retreat

26/8/2018

0 Comments

A week ago we had a code retreat at work. This is essentially a day spent away from our normal work in which we can learn about an aspect of software development.

This code retreat was all about Clean Code and Test Driven Development. The day was ran by Chris Sharp who is an STSM at IBM.

The day was a complete success in my eyes. I've already been pushing for people at work to begin reading the series of Clean Code books and this day re-inforced why it is valuable.

I think there were a number of things that people were quite surprised by:

It is actually quite difficult to develop in a strict TDD way. (people often forget the Red, in the Red, Green, Refactor cycle)
The outcome of the code was much simpler and a lot of people agreed that they likely wouldn't have reached that outcome without TDD.
Software Development is a craft. We tend to get used to churning out code and it can be easy to forget that practice is important.

So those are just a few of the notes that the group had after the session.

Clean Code day to day

I really want to get clean code and TDD brought into our every day work so that we can all benefit from it, although this is going to be a challenge.

I am going to have to lead by example with this, reinforcing it in my scrum's social contract and showing the results.

Finally we are having a second code retreat as only half of the Lab got to go to the first one. I have put myself forward as an "Advocate for Clean Code" in which I will help Chris run the session by going round and giving people guidance whilst they are performing the Kata's.

Thanks :)

0 Comments

Back to it... CMake again!

27/1/2018

0 Comments

So I've decided to revive the JBEngine Project. I gave it up a while ago due to lack of motivation when trying to make it cross platform. The main reason for that was the steep learning curve for using CMake and generally having no real milestones for a very long time. The project was in a state of failing compilation for so long that I ended up giving up. So how am I going to be able to motivate myself this time?

Agile, and how it's effected me

So being a Scrum Master at work for the past work or so I've found out how important it is to break tasks down into small manageable goals/stories. In doing this I will be able to keep myself motivated by ticking things off in my task tracking application. There are still going to be a few problems getting started though.

Too many changes!

So during my development almost a year or so ago I had my project in a state where there were many files edited/added such as many CMake files and lots of changes for making the source code cross-platform. Due to my frantic approach at trying to get the project fully working I now have multiple machines with various files edited with no way of knowing which are required, which aren't working correctly and which I need to just get rid of.

Step by Step Plan

I'm going to take things step by step now, just to get the project in a clean state before making any major changes.

I need to figure out which source code changes are really required and get them checked in.
I then need to figure out which of my CMakeLists actually work and get the correct ones checked in.
I will then delete anything else that is not required.

I want 'git status' to tell me that my repo is completely clean before I even consider making any new changes.

That's all for this time,
Thanks

0 Comments

Web Applications and Leading a Team

7/5/2016

0 Comments

Okay so it's been a while since my last post and I was going to say that the reasons why I've not been doing anything is because I've been busy at work. While this is true I thought I would take this blog post to be more constructive and delve into what I've been learning/developing whilst at work.

The Application Design

At work I have been the lead developer for a small web application that is going to be used internally to help customer support. I have been responsible for creating the design document, discussing with other developers/stakeholders to develop the requirements and iterate over the design to come to a final solution.

Initial Development

In the early stages of the project I was a sole developer working on the application, I chose Flask (a Python web micro-framework) as the tool for the job. Flask is a great tool for creating web applications, as it takes a lot of the complexities away whilst not being as all encompassing as Django.

As Flask is a micro-framework it has a vast number of extensions that allow you to add extra pieces of useful functionality into your web application. Here are a list of some of the extensions I have used:

Flask-SQLAlchemy (provides better integration of SQLAlchemy (a Python DB Abstraction Layer))
Flask-Login (provides user based logins and also deals with session management)
Flask-Uploads (provides easier file upload functionality)

As the project started to take shape I was given some extra development resources that I would need to lead and give direction to so the project could progress.

Team Leading

Looking back I didn't do very well as a team leader at the very beginning of the project. I had various problems early on which I now know ways in which to deal with them:

Problem 1: Communication/Lack of Direction

Initially I didn't give enough direction, forgetting that they hadn't been involved in the early development and didn't understand the reasons that some of the design decisions were made.

This lack of direction was made much worse because I was mainly communicating via e-mail or Instant messaging. This ended up with a lot of back and forth and a lot of frustration on both sides.

After getting some guidance from my manager I decided to set up regular conference calls to actually talk to my teammates in Pune and this ironed out a lot of the issues and also freed up time to actually get some development done.

Problem 2: Python Experience/Development Guidelines

I decided to use Python for the development as there was a lot of other Python developers in the lab which would allow me to seek guidance when I needed it. I didn't take into account that other developers would be working on the project and that they might not have Python experience.

This lack of experience from my teammates was rather difficult to overcome and there were a lot of large code reviews early on explaining exactly why we do certain things, like string formatting for example:


print "This is my string with a variable: {}".format(my_variable)

print "This is my legacy string with a variable: %d" % my_variable

These small differences along with issues with different people using different tab lengths and not using space-tabs caused the reviews to grow a lot.

After doing a few of these code reviews and also talking to my team via the regular conference calls these issues soon ironed out. I also made it mandatory that all my team must run pylint code analysis before submitting for review. In future I will ensure that all my fellow developers have a style guideline or code analysis application like pylint from the start.

Problem 3: Code Review Turnaround/Time Difference

As I was having to do so many large code reviews early on it meant that these would often take me a while to do. I started off doing these code reviews in the morning, as Pune are five and a half hours ahead of us this meant that they were already half way through their next day when I would get to doing the code reviews.

I started doing code reviews as soon as the team had them for me, this would mean I could get it reviewed and comments back to them to address in their morning before I got to work. I would then use evenings whilst they were offline to do my own development work.

This was only a small change but it made the team much more fluid and allowed us to get more tasks done sooner.

Conclusion

So all in all I've learnt a lot, I've now got more skills in Python, Flask and various other frameworks. What I think is more important is that I've learnt how to lead a team much better and I think any future projects would go much more smoothly now that I've learnt these lessons, even if I had to learn them the hard way. I still think I am far from a great leader but I'm a lot closer than I was before this project.

Cheers :)

0 Comments

Development Blog

Timeouts

Stateless (Hey it worked out)

Approach

The Idea

Finished Product

Tech

Python Flask REST API

Multithreading

Stateless-ish

React Frontend

Search and Filter

Infrastructure

Docker

Lessons Learned

Architecture

Containerisation

Improvements

Async or not to Async that is the question.

Flask Restplus

So what are mood marbles?

Basic Concept

Clean Code day to day

Agile, and how it's effected me

Too many changes!

Step by Step Plan

The Application Design

Initial Development

Team Leading

Problem 1: Communication/Lack of Direction

Problem 2: Python Experience/Development Guidelines

Problem 3: Code Review Turnaround/Time Difference

Conclusion

Archives

Categories