Return to -
Return to -UpdateIconReturn to -
Return to -
Return to -Return to -Return to -Return to -

Raising the treasure of dx cluster data

Summary of the dxers' discussion in the email reflectors

Volker Grassmann, DF5AI

December 03, 2005

Introduction

In November 2005, I have launched the following email in the VHF-DX-DISCUSS (blacksheep.org), VHF (w6yx.standford.edu) and in the VUSHF (yahoogroups.com) email reflector. See the discussion which has developed from it ...

Email from Volker, DF5AI, November 23, 2005:

Dear fellow hams,

Very few organizations operate a world-wide data network which can communicate and distribute actual observation data within seconds. Radio amateurs have this type of data system, i.e. the dx cluster network which distributes dx observations spotted between 135 kHz and 10 GHz, 24 hours per day and 365 days per year. In my view, the dx cluster sysops and operators are fulfilling a brilliant and an important job and we all have reason to be grateful to them for maintaining the system at an impressive high service level. The dx cluster's purpose is reporting actual dx information but it does even more: the dx cluster data implicitly reflects all types of radio propagation phenomena allowing a large variety of studies, analyses and, perhaps, forecasts and predictions. Thus, all what we need to do is some sort of data mining by analysing the material systematically.

There are two major obstacles though: 1) raising the treasure of dx cluster data is extremly difficult because of the data format which does not support computer-aided analyses and 2) permanent spam messages ruin the dx spots on daily basis, see the two discussions at the bottom of http://www.df5ai.net/Material/articles6.html

Although no progress was yet made in improving the situation, there are indeed many radio amateurs sharing the same view on this subject. I am aware that RSGB's Radio Propagation Committee has discussed this issue too and I had the opportunity of discussing the subject in some more detail with Dave, G1OGY, just recently. In my view, there is a need to generate some more awareness on the importance of the dx cluster data.

Therefore, you are requested to participate in some sort of brainstorming here. Please address one or all of the below questions and let us know what ideas you have. Perhaps we can generate an attractive package to convince the right people at the right position to optimize the dx cluster network here or there.

  1. What type of data analyses would you recommend assuming the dx cluster data would perfectly meet all requirements in computer-aided data processing? Is there anything we could learn from the data with respect to, say Aurora, FAI, sporadic E propagation?
  2. What type of realtime data analyses and propagation forecasts/predictions do you consider useful and managable? Imagine, we could use actual 144 MHz sporadic E observations to generate an Es-monitor-map of Europe/North America, or so.
  3. What is the right data format in your opinion? Can you provide a proposal explaining the data format of a fictive future dx cluster setup? Please also consider the requirements in data storage (maintaining all dx cluster information over, say 10 years or so).
  4. Distributing dx spots is one issue, what about online chats between dxers that want to arrange dx test runs right on the spot? How would you combine the reporting function of the dx cluster with the chat and talk-back function? Would you keep the two aspects separate or do you think we should have an all-in-one system?

Your thoughts, ideas and comments are very much appreciated.

Email from Graham, F5VHX/G8MBI, November 23, 2005:

Volker, you seem to have missed out on existing software that already performs quite well for two aspects you raise........

with respect to cluster formatting controls/interpretation......you say

> Although no progress was yet made in improving the situation,

this is not strictly true, G7RAU with help from G4FUF and other users feedback has made some good progress with respect to 144 ES plotting/analysis in that ..

  1. there is an agreed format, which is 'forced' on users by some data input checking....the incentive for users to use the package is that in return it provides some useful data feedback
  2. cross checking for validity of data against an on-going database
  3. extensive 'pseudo intelligent' filtering is available
  4. play with Live MUF to experience it all

so for this

> Imagine, we could use actual 144 MHz sporadic E observations to generate an Es-monitor- map of Europe/North America, or so

there is no need to imagine, it exists already, on a worldwide basis, on your desktop, for free........the 'predictive' element is by no means conclusive and trends by many users to be taken as gospel, which it clearly isn't.....

Anyway , several years into this experiment by David , it has become clear that the restricting analysis/ predictive restriction factor is the shape of Europe and the patchy at best input to the system ,

So this is by no means a perfect solution, but it is significant progress...

73 Graham F5VHX/G8MBI etc.......

Email from David, G4ASR, November 23, 2005:

Volker,

As a matter of interest I have been processing DX spots for 10-years now on a daily basis for 6M, 4M, 2M, 70cm, 23cm, 3cm but only from a UK perspective. I have all propagation modes colour coded and the format I've chosen is really good for my purposes - I write the VHF DXER column in Practical Wireless every month. I can easily show trends etc. I've attached a sample. (Editor's note: click button to download the spread sheet: LoadBut)

Cheers,

David G4ASR
RSGB VHF Manager
RSGB Spectrum Forum member

Email from Dave, G7RAU, November 25, 2005:

Hi All

I missed the original posting so I find it difficult to comment on, can anyone forward the the original mail to me so I can input in more detail?

I am breaking a rule I set by mentioning live MUF here (I do not advertise) but since I feel from the discussion that I need to answer a few questions I will answer here and to give a brief history. The live MUF project is now near the end of its 7th year. The idea was (is) to live calculate the es muf based on spots received from the dxcluster (originally in V1 and 2 this was done on text files) and map the data.

The project has involved input / ideas from a number of different hams and thanks to all especially Keith G4FUF who has written / assisted in a large part of the maths for the calculations.I think a number of hams are aware of this as the downloads for V6 and V7 are beyond 5000.

The app works worldwide and the locator data is collected into my data server here in my shack directly from hams using the spot tool to ensure all users get the benefit of uptodate calls / locators.

I do not advertise the app as I feel there is no need and I also do not expect any income from the project even though I host from home the webserver and database server for the project. It is done purely for the reason that I believe this to be a useful tool to fellow hams.

The project was started from a personal interest in sporadic e and was done to assist me in creating a better picture of the realtime es situation, I got fed up with drawing lines on my laminated a4 map of europe. EA6VQ-Gabriel had a big influence in me putting a user downloadable version to the outside world.

The web version of the application is very basic but has had many hits. This was first released from a qrl server but after an excess of 10000 hits per day in the peak of es season I was asked to remove it.

The app only runs on a windows machine but the web version should run in any half decent browser.

I am currently working on the live AU part of the app but qrling 16hrs a day leaves little time for this. Things happen slowly when the brain is pickled from hours on end data analysis at qrl :-)

I also have ~7million processed dx spots spanning 8 years but never time to do a proper analysis of the data...

The app is probably not as user friendly or as slick as it could be but I am not a professional programmer.

Soooo.. I think that there is something better on the horizon from the jist of the comments and I would gladly hang up my boots and use an application that did all the stuff that v7 does, the xyl would

cheer :-)

73 de Dave

Email from Dave, G4RGK, November 25, 2005:

The cluster has evolved over the last 20 years, the original AK1A software is still being used on some RF nodes, some are using CLX some are using Dxspider and there are a variety of other s/ware packages being used. It is run and maintained largely by HF types, if you try and tell them that you want to change the cluster because it does'nt suit your particular purpose they will laugh you out of the room.

The simple solution is to download LIVE MUF from the G7RAU site and spend ten minutes getting it running on a PC you have then got real time Es mapping and the ability filter out absolutely anything you want to. Oh, and by the way it is completely FREE.

Dave G4RGK

Email from Dave, G7RAU, November 25, 2005:

Hi All

Thanks for the replies and mail. I can input some ideas now on the original mail now.

> Very few organizations operate a world-wide data network which can communicate and distribute actual observation data within seconds. Radio amateurs have this type of data system, i.e. the dx cluster network which distributes dx observations spotted between 135 kHz and 10 GHz, 24 hours per day and 365 days per year. In my view, the dx cluster sysops and operators are fulfilling a brilliant and an important job and we all have reason to be grateful to them for maintaining the system at an impressive high service level.

I raise a glass to all sysops / cluster writers and especially to Dirk Hoopman for DX spider.

> The dx cluster's purpose is reporting actual dx information but it does even more: the dx cluster data implicitly reflects all types of radio propagation phenomena allowing a large variety of studies, analyses and, perhaps, forecasts and predictions. Thus, all what we need to do is some sort of data mining by analysing the material systematically.

Well, live MUF collects the data and attempts to work out mode from the spot, this is done real time (well a few milliseconds)

> There are two major obstacles though: 1) raising the treasure of dx cluster data is extremly difficult because of the data format which does not support computer-aided analyses

The DX data is not in a hard to read format. The basic spot shows 2x calls, frequency and a piece of info, a simple string manipulation breaks up the elements and 2) permanent spam

> messages ruin the dx spots on daily basis, see the two discussions at the bottom of http://www.df5ai.net/Material/articles6.html

This is true to a certain degree but it is easily corrected with education. The discussion paper talks about another mode in DXC that handles chat, this already exists, try help chat in the DXC. Also talk is in the DXC to talk directly to a stn across the network.

> Although no progress was yet made in improving the situation, there are indeed many radio amateurs sharing the same view on this subject. I am aware that RSGB's Radio Propagation Committee has discussed this issue too and I had the opportunity of discussing the subject in some more detail with Dave, G1OGY, just recently. In my view, there is a need to generate some more awareness on the importance of the dx cluster data.

Yes I agree, awareness / education is needed.

> Therefore, you are requested to participate in some sort of brainstorming here. Please address one or all of the below questions and let us know ...

Us? This group or another?

> ... what ideas you have. Perhaps we can generate an attractive package to convince the right people at the right position to optimize the dx cluster network here or there.

optimise in what way? The current layout of a dx spot is fine, it is the way it is used and the info it contains that causes a problem. Acually in 27characters you can give all the info needed i.e.

DX de G7RAU: 144300.0 I8MPO IO90IR<ES>JN70FP 1656Z

16 characters are all that is required to give an es spot or

DX de G7RAU: 144300.0 G4RRA IO90IR<AU>IO80BS QTF30/35

Requires 25 to give au info. I only process the 1st 27chr on any DX spot to remove the stuff at the end.

> 1) What type of data analyses would you recommend assuming the dx cluster data would perfectly meet all requirements in computer-aided data processing? Is there anything we could learn from the data with respect to, say Aurora, FAI, sporadic E propagation?

Currently I analyse ES and have started on FAI/AU. already the FAI hotspot map gives a good indication of where possibilities lie.

> 2) What type of realtime data analyses and propagation forecasts/predictions do you consider useful and managable? Imagine, we could use actual 144 MHz sproradic E observations to generate an Es-monitor-map of Europe/North America, or so.

Mapping worldwide is a need I had to address from V4 onwards, it paints a good picture if used correctly. Predicting es probability is harder to approach, as I had said in an earlier mail I have lots of data to analyse in various ways when I have time to look for reliable patterns. AU predictions are partly already supplied by spaceweather.com etc, there is less need for predictive analysis on AU. FAI here is pretty easy to predict from here, MUF>90mhz with the centrepoint less than 1050km away and above 270/90deg. Using mag data increases awareness of two way possibilities.

> 3) What is the right data format in your opinion? Can you provide a proposal explaining the data format of a fictive future dx cluster setup? Please also consider the requirements in data storage (maintaining all dx cluster information over, say 10 years or so).

I have now given the ability for v7 to use ms access/ms sql server/mySQL and SQLite. Oracle could also be done. I store data for es in a different way than other modes. The fields used in the es table are done so to enable instant mapping with no need to recalculate.

> 4) Distributing dx spots is one issue, what about online chats between dxers that want to arrange dx test runs right on the spot? How would you combine the reporting function of the dx cluster with the chat and talk-back function? Would you keep the two aspects separate or do you think we should have an all-in-one system?

The dx cluster does an excellent job for DX spots, I don't think we need a sub system. I already use chat to distribute es alerts on the DX cluster to a group of friends. Chat is very robust.

> Your thoughts, ideas and comments are very much appreciated. A final note: please, do not launch a discussion on the purity of the dx business here - please don't.

I do not understand this? Is this the inaccuracy of calculations? I have found the degree of error a very very minor problem outwayed by the live aspect.

> This would be another discussion which is important too but it is not the purpose of the brainstorming process here. Please don't.

Aha, Well Volker I understand what you wish to achieve and as stated I would gladly hang up my boots. but.. changing DXC format I am clearly against, the backbone is there but users just need to use it more efficiently. I am not sure if you ever looked at live MUF, surprised you are not more informed as the biggest cohort of users are DL. Maybe have a look, you might not like my awful interfaces / config but I believe it will give you an idea what users need as basics, live MUF has been built from many hams inputting their ideas over the last 7yrs.

73 de Dave
Dave Edwards (G7RAU)
http://g7rau.demon.co.uk/

Email from Volker, DF5AI, November 27, 2005:

I apologize for not having answered your emails (which indeed arrived here) earlier because I just returned from a business trip - please forgive me but I wish to summarize all your comments by a dedicated HTML page very soon. By the way: the original posting from November 23 has been distributed by the VHF-DX-DISCUSS email reflector. At least, I have received it, i.e. it is not clear to me why others haven't - this is indeed strange.

I am aware of Dave's "live MUF" project, of course, and I should have mentioned it in my posting - Dave, I am sorry which means: I owe you a nice fresh pint of beer (Guinness?) whenever we will have the chance to meet in person. Dave and I, we had email correspondence about model calculations long time ago because I have used a similar model in the 1980s, see the bottom of http://www.df5ai.net/Material/articles.html - that paper is written in German and is therefore less significant from your perspective but it already considers dx observations as input material for predictions purposes ... i.e. I have ignored even my own paper which may document the fact that the ignorance of the "live MUF" project has no deeper meaning, not at all. Thus, the "live MUF" project indeed reflects an existing prediction tool dealing with dx cluster data ... which is a brilliant initiative!

However, I am no longer happy with my model because it is simply incorrect, i.e. it predicts electron number densities which do not exist in the terrestrial ionosphere (the same comment applies to the live MUF calculations, as far as I can see). There is only one reason which may justify we all continue with this imperfect setup: we currently do not have a better model in place. Implementing a mathematical sporadic E model acceptable from a geophysical perspective still remains a challenge in amateur radio propagation studies.

Before launching a summary HTML page, I hope others will comment too (please refer to the questions in the original posting). I love this type of conversation in the email reflector and would very much appreciate to learn even more thougts and ideas about present and future usage of the dx cluster data.

Email from Dave, G7RAU, December 2, 2005:

Hi Volker and group

I have a few more comments to make.

> I am aware of Dave's "live MUF" project, of course, and I should have mentioned it in my posting - Dave, I am sorry which means: I owe you a nice fresh pint of beer (Guinness?)

No problem

> Dave and I, we had email correspondence about model calculations long time ago because I have used a similar model in the 1980s, see the bottom of http://www.df5ai.net/Material/articles.html - that paper is written in German and is therefore less significant from your perspective but it already considers dx observations as input material for predictions purposes ... i.e. I have ignored even my own paper which may document the fact that the ignorance of the "live MUF" project has no deeper meaning, not at all. Thus, the "live MUF" project indeed reflects an existing prediction tool dealing with dx cluster data ... which is a brilliant initiative!

My german is awful so I have not read it. Yes I agree the model used in live MUF has some problems but these I believe are only due to the lack of information to correctly calculate the possibilities. One of course is the unknown factor over the reflection point on how high this could support reflections and how wide the area covers and what antennuation occurs on the outer edges etc etc. One solution is to have many echo sounders dotted around but these are expensive to run so we only have a limited number in europe and for the mass of europe these are very often not much help unless they are close to the centre.

The other thing that would help is accurate reporting of signals. Keith and I discussed this and we feel this will never happen in the amateur circle. Also we discussed the accuracy of the model in use due to erp considerations but these are not well known and certainly it would take a mammouth effort to gather this data.

> However, I am no longer happy with my model because it is simply incorrect, i.e. it predicts electron number densities which do not exist in the terrestrial ionosphere (the same comment applies to the live MUF calculations, as far as I can see). There is only one reason which may justify we all continue with this imperfect setup: we currently do not have a better model in place. Implementing a mathematical sporadic E model acceptable from a geophysical perspective still remains a challenge in amateur radio propagation studies.

As far as I have found Volker the model works very well most of the time. To actually get pinpoint accuracy with all the factors including the ones I mentioned is really beyond me but as I said if you feel you can improve things then please do. I feel though that aiming too high intially with no spare time will be much more than a lifetimes work and potentially by then there will be little or no interest in the mechanisms behind this and other modes. The use of vhf is changing rapidly..

> Before launching a summary HTML page, I hope others will comment too (please refer to the questions in the original posting). I love this type of conversation in the email reflector and would very much appreciate to learn even more thougts and ideas about present and future usage of the dx cluster data.

I am currently rewriting V7 behind the scenes to enable me to enhance the app quicker. There will be a spot import tool to pull in files of DXC format directly (and custom files). This was a seperate app I wrote a while ago so hopefully this will be in the app before Christmas. I will continue to work when I can on V7. We know the inperfections but I know how many openings I have been sitting in the shack at the right time / right qtf due to the es build up noted on the app. The only thing is I still haven't got an application to persuade my boss I would be better at home than in work for those tearful occasions. :0)

If anybody is interested in processed data then please send me a letter via post and I will reply with a DVD/CD. the db format will be mySQL (currently ~1.5gb but it should compress onto a cd). Also please consider sending me your electronic logs. I can then add these to the base data. I would need date/time, frequency, DXCall and yours and preferably mode of propagation. I can read most db/text formats so this is a real option to build on the already 7 million processed 'spots' in the central server currently going back to 1997. I cannot promise I will process data immediately but I will do it as soon as possible.

Soooo, on a final note Volker, yes we some means currently of looking at previous data, even now I am test importing my 25000 qsos over the last 12 years and I feel if we all rallied together and got this data from logs and some devoted time to analysis we would at least see what patterns / information we can glean from stuff already available to us. Maybe then the penny will drop at some of the 'mysteries' with a detailed investigation of the dat.

73 de Dave
Dave Edwards (G7RAU)
http://g7rau.demon.co.uk/

Email from Volker, DF5AI, December 3, 2005:

Hi Dave,

hi fellow hams,

What can I say? I simply aggree to all your comments including the practical requirements when dealing with mathematical models.

Demanding a more sophisticated sporadic E model in ham radio does not target, say an academical interest - I simply wish to have something useful similar to Booker's model in field-aligned backscatter which can provide a wealth of information in Aurora and FAI dxing without referring to actual data from geophysical observatories. I wonder if we can achieve a similar useful/practical tool in sporadic E too.

Thus, I am just reflecting about a wish list in amateur radio propagation studies: in Aurora/FAI I am quite happy already, in sporadic E we are not yet done - the "Live MUF Project" points in the right direction though. What else? TEP! With high interest, I noticed all this intriguing reports about trans-equatorial VHF propagation (see the TEP email reflector at YahooGroups.com) and I wonder what type of systematical features still await its discovery in the data. Many people argue there is nothing new in ham radio - I disagree because there are many issues worth to be examined in more detail.

With respect to the 25.000 dx QSOs you have filed in recent years, I wish to motivate you to launch a data mining initiative. We all wish to learn what you can see in this impressive amount of dx data ... keep going, please.

TOP

From: http://www.df5ai.net

Copyright (C) of Volker Grassmann. All rights reserved. The material, or parts thereof, may not be reproduced in any form without prior written permission of the author.