Category Archives: web

Calculate wind from thermals

Category : python shell script web

This post is a followup to the last one about Paragliding data gems.

We have collected lots of flights and their GPS location data. From this, several million thermals were extracted and shown on a heatmap. A step forward is to classify these thermals into meaningful groups. Some parameters are easy to extract. For example:

  • Time of the day
  • The month of the year
  • The year
  • Change in altitude
  • Vertical velocity

I have shown in the previous post how the time of the day affects where a thermal is located. The other parameters are also nice to play around with.

One very interesting parameter would be the wind situation. As every pilot knows, the wind plays a crucial role in where turbulences occur and good conditions can be expected.

Finding the wind in our data

The wind consists of two values: Windspeed and direction. And because both values are surprisingly different depending on the height, the wind could be calculated several times. For the sake of simplicity, I am only calculating one wind speed and direction per thermal for now. This could be improved in a later version, when we know there is enough data to fine-grain the selection even more.

How can we calculate wind speed and direction without having access to wind stations and only based on the GPS track? It is not an easy task because the aircraft can turn in any direction and slow down/speed up. Let’s have a look at how a typical thermal with wind looks like:

You can see part of a flight track starting on the lower left and ending on the upper right. There was significant lift and also significant wind drift to the (south-)east. In this case, simply comparing start and end would provide a good estimation of the wind speed and direction.

However, we can not be sure that the pilot follows the wind. Many other scenarios are possible, for example pushing towards the wind while being into several smaller thermal areas. Have a look at this thermal:

Was this change in location triggered by wind drift or by the pilot’s recentering into the core? We can not know by comparing the entry location with the exit location. But we can compute the speed values and direction for each two GPS points. Plotting speed and bearing for another thermal with strong west-wind gives the following:

Here you see speed differences between ~20 km/h and ~60 km/h based on the bearing. The highest speed is ~120° and the lowest ~270°. If a pilot would steer against the wind for a longer time, these values wouldn’t change. We would merely see more points close to the existing ones in a certain area.

The plot above can be improved visually by using polar coordinates:

Here we have the bearing values mapped around a circle and the distance from the middle is the ground speed. If there was no wind at all, this should be a circle around the middle. In this case above, we can estimate a circle and the center gives us wind speed and direction, while the radius of the best fitting circle would be the aircraft airspeed.

The circle can be approximated with some scientific help of Dr. Koch and the mind-blowing scipy.optimize.leastsq function.

Map it on the heatmap

So let’s calculate this for lots of thermals and see how they are affected by which wind:

First of all, you can see a lot more red for the eastern wind than for north(-east). This could be either because the eastern wind is better or because this wind situation is more likely to happen. The heatmap does not reason about the data but the fact is that there are way more thermals appearing with an east- and a west-wind than all the other directions. The database shows how significant that is:

There are way more thermals drifting to the east or west than for any other direction.

When moving closer to a certain area, we can see how frequently they are flown based on the wind. For example, the Wallberg is a well-known west wind mountain:

North- and south-wind show very little thermal action. Even the east-wind does not come close to what happens with west-wind. For Wallberg, this states the obvious for experienced local pilots. However, it might not be obvious for beginners. And it can be helpful to understand which cross country routes work with which wind direction. Does the Baumgartenschneid (across the valley to the north) work with north-wind?

Wrap up

We can now calculate windspeed from GPS fixes and use that data for filtering and grouping. The calculation is compute-intensive and we keep only one value regardless of the height.

Keep in mind that this data is based on wind calculations from the raw thermals. It is not necessarily the same as the overall wind of the day. And there is no statement about turbulence or difficulty. All we can see is that someone found a thermal lift that happened to be in the computed wind drift. We don’t know whether the drift was caused by a valley wind, some lee side rotor, or the overall weather.

Have a look at the Wind based thermal heatmap for yourself.

Related read: Calculating wind speed from the GPS track

Parameters used:

  • Only thermals with a wind speed >= 5 km/h are shown
  • Each thermal has a minimum altitude gain of at least 300 m
  • Wind directions are grouped into [N, NO, O, SO, S, SW, W, NW]
  • Data is based on ~600.000 thermals
  • Public flight database of DHV, mostly from the german community

Technology used:

  • Python with pipenv
  • PyDev
  • Folium, Numpy, Scipy

Paragliding data gems

Category : python shell script web

Paragliding is my beloved hobby and besides offering stunning views and perfect days outside, it also provides a huge amount of flight data to process and play around with. Sites like xc.dhv.de, XContest contain millions of flights from thousands of pilots. These documented flights are gems of data waiting to be investigated by algorithms.

A recorded flight of 4 hours started in the Stubai valley, going up to 3800 m in height, flying to the Öz valley and back.

Outsiders to this sport tend to believe it is about getting up a mountain and simply glide down from there. This might be true for the first few flights, but it can be so much more than that. People stay in the air for up to 12 hours and cover distances of over 500 km. This is possible for the same reason birds can fly such long distances despite the tiny supply of energy they have in their bodies. It is the thermal activity of the air mass.

We all know the sideway wind on the ground. But the vertical winds can be just as strong as a horizontal breeze. And the upward winds are used for climbing as high as possible (and safe) and then glide to the next thermal.

Many books have been written about which factors are important for upward winds and where/when you can expect them to be optimal. But with lots of flight data at hand, it should be possible to see this by example. They just have to be processed and visualized in the right way. This is what Dr. Maximilian Koch and I worked on over the lockdown period.

Get the data

We started with simple scripts for downloading flight data, including all the geo coordinates. For now, it operates on a limited data set because scraping more than a million flights takes time and the process needs to be evaluated first. We don’t want to spider everything and then figure out that something is missing or handled incorrectly. The following examples are based on ~36.000 flights, with a paraglider or a hang glider.

All the data is processed at first for extracting basic information like the takeoff site, the date, and the pilot’s name. In a second step, all the geo-coordinates are processed for thermal activity. I am using this igc-lib for parsing the flights. Judging from some examples, it is not perfect but works well enough on big data sets.

It evaluates the geo-coordinates and extracts thermals and glides from the flights. This information is stored in a local database. Each thermal contains information about the start and end time, the height at both times, the vertical velocity, etc.

Thermal heatmap

Thermal heatmap based on ~36.000 paragliding and hang gliding flights

What can we use that data for? An obvious use case is to show all the thermals on a map, as in the image above. You can see the typical flying routes marked in red. Areas with lots of data appear completely colored, but this is only because the zoom level is so high.

Thermal heatmap zoomed in. The dots appear close to peaks, only few are above the valleys.

When zooming in further, we can see in more detail where to expect upward winds. As the theory states, it is mostly above the peaks and ridges. So far this is similar to other work in the same direction. For example, the paragliding maps show a similar pattern.

Our basic heatmap can be seen and navigated here.

Time based activity

One factor that changes thermal activity is the time of the day. Depending on the sunlight, different areas of the ground are heated and generate a warm airflow.

Therefore, it is interesting to see how thermal activity changes during the day. Here is a time based heatmap in which you can step through all thermals of the day on an hourly base.

The hourly information can be useful to see when and where it is possible to launch in the morning. For long-distance flights, you need to start as early as possible and gain height.

As mentioned before, the data is not complete and it will always have a bias. There are certain areas and routes which pilots typically take. When stepping through the hours, these routes are made visible. Those routes are often used because they are the best possible options. Therefore, even the limited data set is useful for flight planning as they show the most relevant information on the map.

Fun facts

In this dataset of ~36.000 flights, the strongest thermals from start to end have an average climb rate of 7 m/s. There are some outliers showing more than 10 m/s, but all of them can be explained by hardware issues at the start of a flight. These 7 m/s are an average for the whole time of climbing, so there were seconds of a stronger climb as well.

The maximum height gain is 2282 m to an exit height of over 4100 m. This height was reached with an uplift of just below 2 m/s. Within the 36.000 flights, more than 200.000 thermals with a height gain of at least 100 m are reported. So there is an average of 5,5 such thermals per flight. The number is probably much higher in summer than in winter.

Next steps

The data set is still very limited as mentioned a few times. So one goal is to improve the data and download more flights from the respective sites.

There are other interesting questions to ask and possibly answer:

  • Can we calculate the wind conditions out of the tracks?
  • How does it affect thermal activity?
  • Is there something interesting to see in the glides? Can we figure out the best gliding routes, maybe based on other factors?
  • Is it possible to make the data more relevant during a flight? For example, a pilot would only be interested in thermal data that can make him reach a higher position. At the same time, only thermals that can be reached from the current position are of interest.

Can you think of more interesting questions that might be answered by that data? Send me an email and if it is easily possible, I can have a look into it.


Make your website or web-app offline available

Category : Android web

Android developers vs. web

One great advantage of native apps over web apps is that they don’t depend on an online connection to start. Therefore the startup is fast and you can use them in no-network conditions. Just, web apps can also have this advantage when done right.

If you look at a website like Google Docs, you notice that it appears even when you are offline (given that you have visited the same page before). It is even possible to edit files while offline. You can achieve the same with an HTML5 feature called the Offline Application Cache.

Use the Offline Application Cache

While keeping a state locally and syncing requires more effort, making your web(site/app) offline available is easy. You just have to use the Offline Application Cache. This is a single file with information about everything the client should keep in its local cache.

At first you create a file called cache.manifest with the following content:

CACHE MANIFEST
# Cache version #1

CACHE:
/index.html
/css/style.css
/images/header.png
/images/footer.png
/images/background_portrait.png

Change the resources below CACHE: to the required files of your project. Keep in mind that these resources are not requested again, even if they changed. If you want them to be re-downloaded, you need to change the cache manifest itself. This is the reason for the version counter. Increase it by one to make the clients refresh all resources.

The next step is to add it to your site html tag:

<html manifest="cache.manifest" type="text/cache-manifest">

Save, refresh the website on your client and you now have an offline available app. You can test the Offline Application Cache by switching off your server or internet connection and refreshing again. It will reload despite having no connection.

Network and Fallback

In case you have more dynamic content there are two sections you can use in the Offline Application Cache file called NETWORK and FALLBACK:

# Download default_state.xml to local cache
CACHE:
/default_state.xml

# Resources that have to be downloaded from a server
NETWORK:
/current_state.xml

# default_state.xml is used when current_state.xml is not available
FALLBACK:
/current_state.xml /default_state.xml

In this case, current_state.xml requires an online connection and can not be cached. default_state.xml will be added to the cache and used as a fallback when current_state.xml could not be downloaded.

For example, instead of your state data you can put a “state could not be retrieved” message into the default_state.

Wrap up

It is simple to make your web app offline capable. Most of the hybrid- or web-apps I see on the market fail to work without an online connection. It is a pity, because there is so little work necessary to greatly enhance the user experience.

Dynamic state is a different thing, though. Keeping and syncing state is a hard topic, whether native or on the web. While not simple, it is possible with HTML5 Local Storage.

Still, showing your own website with a message is much better than the default browser error. If you have web parts in your application that come from a remote server, be sure to use the Offline Application Cache, at least for the front page and your resources.

A Beginner’s Guide to Using the Application Cache

HTML Standard Application caches


This website switched to HTTPS

Category : web

My webhoster 1blu has finally added the possibility to get a free LetsEncrypt SSL certificate for this website. So www.ulrich-scheller.de is now available via HTTPS.

Let’s Encrypt

With LetsEncrypt getting an SSL certificate is free. There are many reasons for using HTTPS but hardly one against it. This is especially true whenever you enter credentials for logging into your website. In case of this blog and profile page it didn’t matter too much, because the only person with a login is myself. All the content was and is available publicly, so there is no need for a high security level. However, HTTPS is also one of many ranking signals for search engines. And it is just good practice to use encryption wherever it doesn’t hurt.

Simple but not that simple

Moving your website over to HTTPS is very easy. Usually, you just tell your webhoster to get and install a certificate. Or, in case you are hosting yourself, you have to do these steps on your own.

However, keep in mind that this doesn’t magically fix all the problems. After you got your installed certificate and HTTPS running, there is more to check:

  • Links on your website should not point to HTTP urls. Make sure any internal link does not start with “http://”. The best way to fix it is not to change http to https, but instead leave out the complete protocol & domain part. This way links will still work, even if you would move back to http or change your domain. You can use www.whynopadlock.com for checking the links on your site.
  • Change the url of your Page in WordPress to https://
  • When another blog linked to your page, this link will continue to point to the HTTP version. For search engines, your HTTP website is different from the HTTPS site. To prevent losing a good ranking, make sure to redirect from your HTTP to the HTTPS version. This also ensures your visitors will use the secure protocol from now on.

There can be much more, depending on your setup. Read through this detailed guide to see what else might apply in your case. WordPress and a professional web hosting simplifies much of that work. However, most of the time it also limits how far you can go. For my page, I had to wait multiple years until Let’s Encrypt became available. Features like HTTP2 support are still not available.

Now I need to go ahead an fix the links on this site. If you see anything else that is breaking encryption on my site, please tell me in the comments.


Recent Posts

Tweets