So I've just listened to the Talk Python Podcast for digging into visual studio code and been really thinking about how useful this could be to me from day to day. I already use vs code for all my python and robot framework development I do at work and home so it's already part of my workflow. 

So the workflow for development at work is this:

1. Develop on code locally using Eclipse
2. Push changes to the build server.
3. build on the build server.
4. pull down any compiled files to local machine if required.
5. package up the code on the build server.
6. push the package to a piece of hardware for testing.

This whole process could be much simplified with points 2) and 4) if I could simply develop on the build server using the vs code remote development tools. Not only 2) and 4) would be much improved but since you can actually use the terminal built into vscode to operate on the filesystem of the remote machine then I could also run the make commands for doing the whole build and packaging process as well.

This seems ridiculously powerful!

On my home machine I have a separate installation of Linux Mint for doing all my python development on, this is for a couple of reasons:

1. I prefer developing python on Unix based systems as I have full access to the bash shell and installing python and working with packages is much simpler there.
2. Not many video games have installs for Linux so it keeps me productive :)

Well this won't help with number 2 but It will certainly help with number 1.

So Windows Subsystem for Linux seemed really good apart from you can't use any Editors with the WSL files... until now!

I would no longer have to work in vi/vim if I wanted to do any python development in WSL, instead I can use the remote development tools of vscode to have the same workflow as I would on a unix based machine. This would make using WSL on windows a possibility for me and save me some much needed space on my SSD as I will only need one OS :)

A while ago I posted a blog post about a hackathon project I did for work. In this project it required me to run multiple SSH sessions to many machines at once to query the state of the system.

Recently I've been listening to a lot of the Talk Python To Me Podcast, and hearing all about the new async features of Python 3. This got me thinking about this project and how this would be the perfect candidate for some async functionality. Why is this?

So we had a fairly large number of machines that we were wanting to run these SSH Commands on and many of those machines were simply not turned on or currently unavailable for SSH connections. This caused an issue as we simply had to wait for the timeout to occur which was down to about 1 second to ensure we gave enough variance for slow network.

During this 1 second window each of the worked threads were simply waiting there counting down the milliseconds. This was extremely inefficient!

So Python introduced the async and await keywords but unfortunately it is likely going to be a while before there are many ORM's that are async capable. This is really handy as I had decided that the API was going to be fairly stateless other than the in memory cache, which means implementing this new approach should be fairly simple.

So I used Flask for the web framework for the API and there is now a drop in replacement being developed called Quart. This enables the async and await functionality into that same Flask development style that everyone loves with Flask.

​Along with Quart is another python package called asyncSSH which allows the use of async and await for SSH Connections.

So the actual changes required to test out this new approach hopefully shouldn't be too much effort.

The big question is will this actually improve performance?

At work we occasionally have a Hackathon day where everyone in the lab is able to work on any project they like whether that's just practising some Test Driven Development(TDD) or creating something that could help with your day to day work.

So for this Hackathon I got together with Adrian Moldovan and Tom Fletcher to work on a project that would hopefully address a major issue we have at work.

Our problem is that our product has many different platforms and we don't have all the different platforms in each of the teams so we often have to share our hardware. The problem with this is that in juggling the hardware between teams some hardware can often get lost along the way.

The major problems we have are:

• Finding the a specific platform that you need to test.
• Finding hardware on a specific code level.
• Finding hardware that is in a good state for you to borrow.
• Figuring out which hardware isn't being utilised.
• Finding out who you should talk to about borrowing a piece of hardware.

As you can see we have quite a variety of kit on many different levels and this information is invaluable for quickly finding the piece of hardware you require. 

So let's talk about the tech that has been used for this project. Firstly I've been reading a lot about microservices and was intrigued by this concept of splitting out what would normally be on large application into smaller portions. By no means is this a microservice based architecture for this project as it lacks some of the required infrastructure to be that way.

This project is split into two smaller pieces:

1. A Python REST API which performs the same ssh command to multiple machines at once and combines the result into a single response.
2. A React Web Frontend which displays a table of information for multiple machines.

So the basic idea for the project was to run many of the same ssh command to all of the machines and get a point in time capture of all of the state required to solve the above problems. To achieve this we decided to use threads to be able to kick off multiple ssh sessions at once.

I tend to avoid using multi-threading whenever I can because it massively increases complexity but in this situation it seemed like the correct thing to do.

So I really liked the idea of the REST API being a really simple worked that simply ran commands on multiple machines and output the result.

I decided that we should try and keep this fairly stateless and so the only real state that the API has is a simple in memory cache which wouldn't even be necessary if it wasn't for the sheer number of machines that we are trying to perform the commands on.

The caching itself is only held for 15 minutes for each machine to ensure the data isn't too stale.

So this portion of the project is where all the shiny things happen, without this the user would need to be able to easily read and interpret JSON data which is not a skill many people have.

For ease of use we added a search functionality along with some filter buttons which will allow the user to quickly narrow down the choice that they are after. There is also a refresh functionality which will call to the API and tell it to do a full refresh not including cached data. This is mostly for people who really need the most up to date information.

For this project we really wanted to try out using Docker as we wanted to be able to deploy these two applications as simply as possible. After all this was a Hackathon and so it is a perfect opportunity to try and harness new skills.

Both projects were there own separate containers and the frontend simply needed an argument on run which passed in the address for the API.

I think the biggest things I learnt were to do with application architecture specifically how this style of application level separation of concerns allows for extendable and reusable functionality. The API itself can be used by any frontend client. If React is no longer the flavour of the month then a new frontend could be made, or if someone wanted this functionality on mobile then they could still simply call the same API.

This whole idea of containerisation I now see if incredibly powerful, it is much easier developing on the same platform that you are deploying to and also this ability to deploy anywhere that has just the one dependency of being able to run Docker is so useful. 

I will definitely be doing a lot more with Docker in future.

There are always improvements that can be made, specific ones I would like to look into are:

• Support for the OpenAPI /Swagger API Specification.
• Adding Docker Compose support for deploying both of the Containers together.

So the past few days I've been working with CMake to try and get my some of the JBEngine Libraries running for Linux. The first thing I did was use vcxproj2cmake tool to generate the CMakeLists.txt from my already created Visual Studio Project Files. To start with I concentrated on the Maths and Sound Libraries as they were the easiest to do. Maths Libraries only requires some third party includes of glm and the Sound Library only has one library dependency on FMOD. 

To get the tool working I ended up having to copy all it's files into the same directory as the project file. It just didn't seem to work with a path. Secondly I then had to manually change all the paths that I had entered (or Visual Studio had generated) for the include and library directories in the Visual Studio Projects. Microsoft tends to default to using "\" for directories where as Unix system use "/". Once I had done this It finally generated a CMakeLists.txt for me, but this was only the beginning!

Being the naive developer that I was a few years ago I had scattered through my source code the use of the "\" for directories instead of "/". This was one of the first things I had to address. It was mostly the includes in the files so it wasn't too hard to track down. This wasn't too time consuming for the Maths and Sound libraries as there are no more that 5 files in each. Doing this for the whole JBEngine Source may be a little more tedious.

The next problem I encountered was resolved by this Stack Overflow post: Why does MSVC Let me do this. I was essentially passing an anonymous object to a function. It was a little confusing to track down as it wasn't a simple example:
​

void Union(Bounds& bound);
void Union(vec3& v);

void Bounds::Union( Bounds& bound )
{
    Union(bound.GetMin());
    Union(bound.GetMax());
}

What you don't see here is that Bounds.GetMin() and Bounds.GetMax() do not return a reference. This means that the returned vec3 object from the functions are newly created and therefore anonymous objects. MSVC automagically resolves this in some way but g++/gcc does not do this for you. This took me a little time to figure out as I had never even had to think about this sort of thing before.

They were the only real code issues that I faced, the rest was all about learning CMake.

So I started off with an already "complete" CMakeLists.txt which would work for Windows. Unfortunately this was never going to work out the box for Linux. Well at least it didn't for the Sound Library. First of all I had to get the Linux version of the FMOD Libs.

I had quite a bit of trouble getting used to how CMake works, specifically with include and library paths and also using the if(UNIX) branches for OS specific target_link_libraries. I finally managed to get CMake working and now I only had to deal with the make compilation issues.

What I hadn't realised is that fmod had changed significantly since I got the library for Windows so then I had to adapt the code to use the new library. 

Once I had managed this I then created a Test application to play a sound. So it was all quite easy I just had problems with:

1. vcproj2cmake
2. code issues
3. cmake issues
4. library compatibility issues.
5. and then more code issues due to the new library.

It really was a breeze.

On a serious note it wasn't half as bad as I thought, and I've learnt much more about CMake. The next things I will need to look at are:

• Adding CMake Dependencies i.e. JBEngine Library requires all the other smaller Libraries such as Sound, Maths, Renderer etc.
• Ensuring I haven't broken anything in Visual Studio whilst messing around with the project files.
• Looking into Visual Studio Project/Solution Generation from CMake.