Category Archives: shell script

Calculate wind from thermals

Category : python shell script web

This post is a followup to the last one about Paragliding data gems.

We have collected lots of flights and their GPS location data. From this, several million thermals were extracted and shown on a heatmap. A step forward is to classify these thermals into meaningful groups. Some parameters are easy to extract. For example:

  • Time of the day
  • The month of the year
  • The year
  • Change in altitude
  • Vertical velocity

I have shown in the previous post how the time of the day affects where a thermal is located. The other parameters are also nice to play around with.

One very interesting parameter would be the wind situation. As every pilot knows, the wind plays a crucial role in where turbulences occur and good conditions can be expected.

Finding the wind in our data

The wind consists of two values: Windspeed and direction. And because both values are surprisingly different depending on the height, the wind could be calculated several times. For the sake of simplicity, I am only calculating one wind speed and direction per thermal for now. This could be improved in a later version, when we know there is enough data to fine-grain the selection even more.

How can we calculate wind speed and direction without having access to wind stations and only based on the GPS track? It is not an easy task because the aircraft can turn in any direction and slow down/speed up. Let’s have a look at how a typical thermal with wind looks like:

You can see part of a flight track starting on the lower left and ending on the upper right. There was significant lift and also significant wind drift to the (south-)east. In this case, simply comparing start and end would provide a good estimation of the wind speed and direction.

However, we can not be sure that the pilot follows the wind. Many other scenarios are possible, for example pushing towards the wind while being into several smaller thermal areas. Have a look at this thermal:

Was this change in location triggered by wind drift or by the pilot’s recentering into the core? We can not know by comparing the entry location with the exit location. But we can compute the speed values and direction for each two GPS points. Plotting speed and bearing for another thermal with strong west-wind gives the following:

Here you see speed differences between ~20 km/h and ~60 km/h based on the bearing. The highest speed is ~120° and the lowest ~270°. If a pilot would steer against the wind for a longer time, these values wouldn’t change. We would merely see more points close to the existing ones in a certain area.

The plot above can be improved visually by using polar coordinates:

Here we have the bearing values mapped around a circle and the distance from the middle is the ground speed. If there was no wind at all, this should be a circle around the middle. In this case above, we can estimate a circle and the center gives us wind speed and direction, while the radius of the best fitting circle would be the aircraft airspeed.

The circle can be approximated with some scientific help of Dr. Koch and the mind-blowing scipy.optimize.leastsq function.

Map it on the heatmap

So let’s calculate this for lots of thermals and see how they are affected by which wind:

First of all, you can see a lot more red for the eastern wind than for north(-east). This could be either because the eastern wind is better or because this wind situation is more likely to happen. The heatmap does not reason about the data but the fact is that there are way more thermals appearing with an east- and a west-wind than all the other directions. The database shows how significant that is:

There are way more thermals drifting to the east or west than for any other direction.

When moving closer to a certain area, we can see how frequently they are flown based on the wind. For example, the Wallberg is a well-known west wind mountain:

North- and south-wind show very little thermal action. Even the east-wind does not come close to what happens with west-wind. For Wallberg, this states the obvious for experienced local pilots. However, it might not be obvious for beginners. And it can be helpful to understand which cross country routes work with which wind direction. Does the Baumgartenschneid (across the valley to the north) work with north-wind?

Wrap up

We can now calculate windspeed from GPS fixes and use that data for filtering and grouping. The calculation is compute-intensive and we keep only one value regardless of the height.

Keep in mind that this data is based on wind calculations from the raw thermals. It is not necessarily the same as the overall wind of the day. And there is no statement about turbulence or difficulty. All we can see is that someone found a thermal lift that happened to be in the computed wind drift. We don’t know whether the drift was caused by a valley wind, some lee side rotor, or the overall weather.

Have a look at the Wind based thermal heatmap for yourself.

Related read: Calculating wind speed from the GPS track

Parameters used:

  • Only thermals with a wind speed >= 5 km/h are shown
  • Each thermal has a minimum altitude gain of at least 300 m
  • Wind directions are grouped into [N, NO, O, SO, S, SW, W, NW]
  • Data is based on ~600.000 thermals
  • Public flight database of DHV, mostly from the german community

Technology used:

  • Python with pipenv
  • PyDev
  • Folium, Numpy, Scipy

Paragliding data gems

Category : python shell script web

Paragliding is my beloved hobby and besides offering stunning views and perfect days outside, it also provides a huge amount of flight data to process and play around with. Sites like xc.dhv.de, XContest contain millions of flights from thousands of pilots. These documented flights are gems of data waiting to be investigated by algorithms.

A recorded flight of 4 hours started in the Stubai valley, going up to 3800 m in height, flying to the Öz valley and back.

Outsiders to this sport tend to believe it is about getting up a mountain and simply glide down from there. This might be true for the first few flights, but it can be so much more than that. People stay in the air for up to 12 hours and cover distances of over 500 km. This is possible for the same reason birds can fly such long distances despite the tiny supply of energy they have in their bodies. It is the thermal activity of the air mass.

We all know the sideway wind on the ground. But the vertical winds can be just as strong as a horizontal breeze. And the upward winds are used for climbing as high as possible (and safe) and then glide to the next thermal.

Many books have been written about which factors are important for upward winds and where/when you can expect them to be optimal. But with lots of flight data at hand, it should be possible to see this by example. They just have to be processed and visualized in the right way. This is what Dr. Maximilian Koch and I worked on over the lockdown period.

Get the data

We started with simple scripts for downloading flight data, including all the geo coordinates. For now, it operates on a limited data set because scraping more than a million flights takes time and the process needs to be evaluated first. We don’t want to spider everything and then figure out that something is missing or handled incorrectly. The following examples are based on ~36.000 flights, with a paraglider or a hang glider.

All the data is processed at first for extracting basic information like the takeoff site, the date, and the pilot’s name. In a second step, all the geo-coordinates are processed for thermal activity. I am using this igc-lib for parsing the flights. Judging from some examples, it is not perfect but works well enough on big data sets.

It evaluates the geo-coordinates and extracts thermals and glides from the flights. This information is stored in a local database. Each thermal contains information about the start and end time, the height at both times, the vertical velocity, etc.

Thermal heatmap

Thermal heatmap based on ~36.000 paragliding and hang gliding flights

What can we use that data for? An obvious use case is to show all the thermals on a map, as in the image above. You can see the typical flying routes marked in red. Areas with lots of data appear completely colored, but this is only because the zoom level is so high.

Thermal heatmap zoomed in. The dots appear close to peaks, only few are above the valleys.

When zooming in further, we can see in more detail where to expect upward winds. As the theory states, it is mostly above the peaks and ridges. So far this is similar to other work in the same direction. For example, the paragliding maps show a similar pattern.

Our basic heatmap can be seen and navigated here.

Time based activity

One factor that changes thermal activity is the time of the day. Depending on the sunlight, different areas of the ground are heated and generate a warm airflow.

Therefore, it is interesting to see how thermal activity changes during the day. Here is a time based heatmap in which you can step through all thermals of the day on an hourly base.

The hourly information can be useful to see when and where it is possible to launch in the morning. For long-distance flights, you need to start as early as possible and gain height.

As mentioned before, the data is not complete and it will always have a bias. There are certain areas and routes which pilots typically take. When stepping through the hours, these routes are made visible. Those routes are often used because they are the best possible options. Therefore, even the limited data set is useful for flight planning as they show the most relevant information on the map.

Fun facts

In this dataset of ~36.000 flights, the strongest thermals from start to end have an average climb rate of 7 m/s. There are some outliers showing more than 10 m/s, but all of them can be explained by hardware issues at the start of a flight. These 7 m/s are an average for the whole time of climbing, so there were seconds of a stronger climb as well.

The maximum height gain is 2282 m to an exit height of over 4100 m. This height was reached with an uplift of just below 2 m/s. Within the 36.000 flights, more than 200.000 thermals with a height gain of at least 100 m are reported. So there is an average of 5,5 such thermals per flight. The number is probably much higher in summer than in winter.

Next steps

The data set is still very limited as mentioned a few times. So one goal is to improve the data and download more flights from the respective sites.

There are other interesting questions to ask and possibly answer:

  • Can we calculate the wind conditions out of the tracks?
  • How does it affect thermal activity?
  • Is there something interesting to see in the glides? Can we figure out the best gliding routes, maybe based on other factors?
  • Is it possible to make the data more relevant during a flight? For example, a pilot would only be interested in thermal data that can make him reach a higher position. At the same time, only thermals that can be reached from the current position are of interest.

Can you think of more interesting questions that might be answered by that data? Send me an email and if it is easily possible, I can have a look into it.


Personal Jenkins Server with Docker

Nowadays, every software development team should have a continuous integration server like Jenkins. This is as true for Android developers as for any other platform. It makes sure the current source code compiles and all the tests succeed, so nobody is blocked by a broken build. A CI also forces you to have a build in one step and to perform it regularly, usually on every commit.

Most often a continuous integration platform is used for development teams. However, it also gives many benefits to single developers. Multiple times I had the problem that old projects would not compile or work after switching to a new computer. Also, I often forgot to run all tests if they seemed unrelated to my code changes. And last but not least there is the deployment-pain when the last time was long ago.

Where to host?

A Jenkins build server would solve all these problems. But I didn’t want to spend a lot of money on hosting, because it is for private, closed-source projects with no profit. And running Jenkins on my development machine does not help, because it is still the same environment as in my IDE. My first idea was to use a RaspberryPi server. While this is not a fast computer, it runs on very little power and would have more than enough time for builds.

After playing around with it, I discarded that option again. Jenkins on a RaspberryPi works – but it is not an x86 device. So if you are not using the Android SDK, a Raspberry might be an option. For me it is not, because the Android SDK is not available on ARM systems.

Another low-cost option is VirtualBox. You set up a linux server inside a virtual machine and host Jenkins in there. Although the virtual machine is hosted locally, it can easily be transferred to anywhere if necessary. I had this option in mind for a while. But I didn’t like the overhead VirtualBox brings into it.

So when Docker announced their MacOS release, I was eager to try it.

Set up Jenkins

Installing Docker is an easy task. There is a good Getting Started guide on their website. You will learn using the docker command and how to run containers. It turns out there already is a working Docker container with an installed Jenkins. Run it with

docker run -p 8080:8080 -p 50000:50000 jenkins

This starts up a new container with Jenkins running on port 8080. You will see the following website:

Docker Jenkins first login

For getting the initialAdminPassword you have to access the running container.

Ulrichs-MBP:~ uscheller$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                              NAMES
82812e782a95        jenkins             "/bin/tini -- /usr/lo"   7 minutes ago       Up 7 minutes        0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   stupefied_bell

will show you all running containers. Look up the name of the container, in my case it is stupefied_bell. Edit the following command with your container name:

docker cp stupefied_bell:/var/jenkins_home/secrets/initialAdminPassword . && cat initialAdminPassword && rm initialAdminPassword

This prints the initialAdminPassword for logging in. If you enter it on the website, you can continue the setup process and have a running Jenkins in Docker.

Useful commands

You can stop your running machine with

docker stop stupefied_bell

When you want to start it again, don’t use the docker run command from above. It would create a new instance. Instead, simply use

docker start stupefied_bell

Get a shell on this container as user jenkins

docker exec -it stupefied_bell /bin/bash

Get a shell on this container as root

docker exec -u 0 -it stupefied_bell /bin/bash

Install Android SDK from shell

Docker Speed

Performance-wise this setup is great. The docker container starts and stops within seconds. The only delay is caused by Jenkins startup-time, showing the please-wait message. My Jenkins is not yet overloaded with plugins and jobs so it takes less than 15 seconds for everything to be available.

I tested the build-performance with my Android project Laska:

gradle clean build test

The time for executing this command on my native machine is around 2 minutes 34 seconds. The Docker container takes only 1 minute 54 seconds. In multiple runs the outcome was always in favor of Docker. I can not explain why this happens, as native should be the fastest. It might be a configuration setting on my machine.

Wrap up

Using Docker to host Jenkins is a great solution for solo developers. It is easy to set up, especially with a pre-packaged Jenkins container, and can be transferred to a dedicated server if necessary. Build speed is the same as in a native environment.


Gain productivity by scripting your workflow

Category : shell script

One thing that helps me a lot during my everyday routine is having shortcuts for every simple action I perform. For example

  • uninstalling a specific Android app
  • opening the current XCode project
  • opening a ‘tail -f’ on the logfile of our server

can be reached with two mouseclicks from the dock. Some of those are nothing more than a copy and paste shortcut, the others are one line of shell script. There is really no magic inside, every decent developer should be able to do that. However, at my current workplace, out of ~20 developers, I have never seen anybody creating his own. So take this article as a reminder for reviewing your daily routine and automate the most mundane tasks.

Because creating shortcuts is too easy and doesn’t deserve a howto, I will focus on scripting actions on a remote server. Be sure to use private/public key authentication, so there is no need for typing passwords 20 times a day. This doesn’t require a Unix-style operating system, it is also possible with the Putty Tools on Windows. With that, you can execute remote scripts on a server.

My most often used script looks like this:

ssh root@myserver.com

Which gives me a logged-in terminal at this server when executed. It is also very helpful to access logfiles with just one click. Just append the remote command to the commandline:

ssh tomcat@myserver.com "tail -f /opt/apache-tomcat-6.0.20/logs/catalina.out -n 1000"

That prints the last 1000 lines of the logfile and keeps appending more lines. You might think of many more useful cases already. For example, restarting a server process, watching the network traffic with ngrep or printing the server load. You can also put these scripts into version control and use them with a team of developers.

Scripts like that saved me a lot of time already, and only take a few minutes to set up. And the much more important benefit is, that I use the available information much more often. If it takes 3 seconds to open the logfile, I will look into it even for small problems. But if it takes 30 seconds of a mundane and repetitive workflow…

 


Recent Posts

Tweets