Using dx cluster data in radio propagation studies
Volker Grassmann, DF5AI
August, 2002, revised February 2005
Very few organisations operate a world-wide data network which communicates realtime data, which records all information for later retrieval, which triggers alerting services and which may be accessed by any user in almost any place in the world free of charge. Radio amateurs own this type of network, i.e. the dx cluster (dxc) network. The dx cluster represents a permanent dx monitoring system between 1.8 MHz and 10 GHz that operates 24 hours per day and 365 days per year on a world-wide basis. The value of the cluster is based on the dx data which radio amateurs in all continents contribute to the system. Each contribution adds a new line to the so-called spot list (see figure 1) which the dx cluster distributes in the packet-radio networks and in the internet every three minutes. In fact, the dx cluster may be considered one of the most amazing institutions in amateur radio and its users are grateful to the dxc operators and sysops for providing and maintaining this fine service on a regular basis.
We may speculate that the dx cluster tracks the majority of dx openings in the world, i.e. its data base reflects almost all elements in terrestrial radiowave propagation. From this perspective, the dx cluster data must be considered a treasure which can support radio propagation studies in various aspects - in fact, this type of data does not exist anywhere else in the world. However, raising the treasure is difficult because the spot list is designed for humans reading the data but ignores the requirements for electronic data processing.
A new discipline in amateur radio: data analyses, number crunching and scientific modeling
Imagine your national met-office would not provide meteorological summary reports but would send you the raw data from all weather stations in your country. Evidently, there is more information than before but the view on the overall weather systems is now missing similar to a puzzle where you cannot identify the picture without assembling the pieces. Referring to the dx cluster data, radio amateurs receive the dx raw data but can hardly identify systematical features in the dx data for the same reason.
For example: the OH9W/OH2AQ dx cluster maintains its data since 1997 (its operators deserve a price for looking ahead), i.e. in a couple of years we may study Sporadic-E statistics covering a full solar cycle. In the field of HF, VHF, UHF and SHF, in tropospheric as well as ionospheric radio propagation, in Sporadic-E, FAI, Aurora, TEP, Auroral-E and other types of terrestrial radiowave propagation, we indeed hold valuable information in our hands and there is no question that many effects and phenomena await their discovery when analyzing the dxc data systematically. In the European email reflectors, just another example, VHF radio amateurs speculate about declining tropo conditions, i.e. many VHF operators got the impression that tropospheric dx band openings occur less often compared to the 1980s and 1990s. Does global warming cause this effect or does this effect result from a decline in QSO activity, or does the effect originate from statistical variations - does this effect exist at all? Very likely, the answer may be found by analyzing the dxc data.
The dxc data may support and stimulate a large variety of applications, analyses, evaluations and studies and we may perhaps open the door to a new discipline in our hobby, i.e. data analysis, number crunching and scientific modeling. We got the personal computers and we got the dxc data, combined with the creativity of radio amateurs we may certainly expect intriguing results, dx forecasts, new opportunities in dxing and other features and facts which could make amateur radio more interesting to existing and to potential radio amateurs as well.
Raising the treasure
What do we need to raise the treasure? The data access is indeed no obstacle: the OH9W/OH2AQ dx cluster operators, for example, have implemented an internet interface to the dxc database and will certainly support full database access if required. Even the total data volume is now obstacle to modern personal computers, see figure 2.
The dx spot list formats the date, the time and the frequency information so that these items may be picked by computers very easily (see figure 1). Hence, everything is prepared, let's start the data analysis right now. We can't! See the problem in figure 1: the computer cannot retrieve the geographical positions from the list, even humans may fail to identify the grid locators properly, if those are available. Unfortunately, the majority of dx spots do not meet the requirements of electronic data processing. What a shame! Very few operators realized this problem by supporting pseudo formats such as 'JQ68TB > JP81EI'. Gabriel (EA6VQ), on the other hand, has created an alternative dx submission form which leads in the right direction but too few radio operators are aware of this feature, unfortunately.
Figure 3 suggests an alternative data format which places the grid locators into separate columns. This concept cannot improve the existing data, of course, but it may support electronic data processing in the future. On the other hand, the list format cannot guarantee the quality of the input data, i.e. there is also a need to review the dx submission form in general. For example: the submission form may notice the user that grid locators is an important information which should be added to all dx spots entered in the system (however, we cannot force the user to do so of course).
Raising the treasure is not that simple, obviously. We are very close to an extraordinary world-wide radio propagation database - however, the final step is missing ... because radio amateurs cannot manage changing the format of an ASCII text file (i.e. the spot list) in the existing dx cluster network. I understand the effort required to make the change - it is however worth to think about it, definitely.
See also the discussion page ...