development.xml: RADIO_LINK statistics messages

[jlpoltrack]

New messages for radio link statistics.

Follow-up on: #343

Done on behalf of @olliw42

[olliw42]

many THX to @jlpoltrack for helping out again!

@peterbarker @hamishwillee This is the 2nd step in the radio link renovation effort, as promised already here ArduPilot/ardupilot#24577.

It adds link statistics, and it does so by logically separating "high-frequency" from "low-frequency" link stats info. The rational is to keep the traffic low and to only send frequently what updates frequently. The result are the two proposed messages RADIO_LINK_STATS (= high frequency) and RADIO_LINK_INFORMATION (= low frequency).

The data contained in RADIO_LINK_STATS is typcially updated with each new ota-frame received. This typically is the same frequency with which new rc data are received and with which RADIO_RC_CHANNELS is emitted by the MAVLink receiver. This message has been discussed to some extend already here mavlink#1920, and it in fact has been the discussion template for a similar DroneCAN message. The message submitted here is slightly further improved. I believe to know that also e.g. ELRS is very interested in such a message.

The RADIO_LINK_INFORMTION message in contrast contains data which in most cases are static, i.e., do not change with time. In some systems they can slowly change, "slow" meaning slower than the ota-frame rate. Examples would be when the mode is switched like in TBS Crossfire, which switches between 150 Hz and 50 Hz modes, or with dynamic power like in ELRS (I did watch quite a number of fpv videos confirming that with ELRS the power usually changes slowly).

The RADIO_LINK_STATS message will be immediately of high use. I do have the code to bring it into ArduPilot in a next PR. Examples of what kind of analyses it can provide you can find here https://discord.com/channels/1005096100572700794/1005096101239603232/1192769117267173376 (and few more in few posts below).

The RADIO_LINK_INFORMATION message will not be immediately usefull, but it's added here so you see where all this is heading to. For instance, the receiver sensitivity can be used to provide a better conversion to MAVLink rssi units, or provide a range estimate like in one of the ELRS EdgeTx widgets (there is a comment by JB in one of his videos saying how useful knowing receiver sensitivity could be). The data rates in the proposed message can help to better throttle the ArduPilot streams. And so on.

The data in either messages should also be useful for improved OSD (see comment here ArduPilot/ardupilot#24577 (comment)).

[hamishwillee]

@peterbarker We did indeed got through extensive discussion of a variation of this in MAVlink some time ago, and also align with Dronecan. From a MAVLink-design point of view these messages look fine - modulo a few typos (receiption).

My concern on this is that I do not have any expertise or opinions about radio links - so I have no way to personally evaluate whether this is sufficient for the broad range of radios we might be interested in. If it is not, or has known limits about application, we should be very clear on that up front so we all know the situations in which use of the protocol makes sense.

@auturgy Could you please have a look? Is there anyone else we need to look at this?

[auturgy]

Probably worth @wvarty or @JyeSmith having a look: in the short term I expect mLRS and ELRS to be the primary beneficiaries. From my point of view I'd like to understand whether link quality, rssi, noise and other metrics related to link margin all need separate fields, or whether a single field plus enum would suffice, shrinking the high frequency message somewhat. It really comes down to whether the intent is for all of that information to be available and useful, or if we're just covering off what might be available. If the latter, let's use an enum.

[olliw42]

I have no way to personally evaluate whether this is sufficient for the broad range of radios we might be interested in. If it is not, or has known limits about application, we should be very clear on that up front so we all know the situations in which use of the protocol makes sense.

first off, unlike with RADIO_RC_CHANNELS; the situation here is better since the message can later be extended ... not saying we should plan for that and allow obvious mistakes now, but it means we do not necessarily have to anticipate the future century

The messages as proposed sure do have limits. If one would try to accomodate all thinkable cases it would become massive and clumsy. For instance, the "low-frequency" message is not well suited for dual-band (like 2.4GHz & 868/916 MHz) systems (note: mLRS supports dual band since a while). The "high frequency" message however is.

I think we want to separate the discussion into two parts, the "high frequency" message and the "low-frequency" message, and related to that the obvious question which field/data should go where. For the moment, I think we should concentrate on the "high frequency" message.

in the short term I expect mLRS and ELRS to be the primary beneficiaries.

in the short term maybe. I don't have such systems but I think also RFD900 and DragonLink have some combined rc channel data and mavlink capability and thus might at some point jump on board too. TBS in principle too, but we don't hear a lot these days from TBS, so what do we know.

I think I have considered ELRS and TBS in my thoughts. The only field where - IMHO - ELRS maybe might have a different stance is the power fields since they do dynamic power, and technically it could update with every packet (I think), but as indicated I watched a number of FPV videos which had OSD and concluded it rather changes not like every 10 ms. Will be good to hear from ELRS directly however. And then there is always the questio of how important it really is.

I'd like to understand whether link quality, rssi, noise and other metrics related to link margin all need separate fields,

Y E S Y E S Y E S. That's - IMHO - really the best part of it. It's hard to convincingly explain in few words, but once you did some analyses like here https://discord.com/channels/1005096100572700794/1005096101239603232/1192769117267173376 you really will appreicate having all three. LQ is certainly the most important from a practical point, i.e., would be what you display brightly. Rssi and noise however are really usefull to check your system, and if it's just if the antenna is well placed.

[hamishwillee]

And then there is always the question of how important it really is.

The question of importance (for all fields) probably depends on who you think is the consumer. What's an OSD going to do with this data? If it is just for display then 10ms is a very high rate. By contrast, if you're analyzing behaviour to perform some action or log then you need a high enough rate for that analysis to be performed either live or as a post process.

Upshot, the rates probably should be expected to be tunable depending on the channel and purpose. That's what the streaming rate setters are for, but perhaps these need to be default values for each radio type.

For instance, the "low-frequency" message is not well suited for dual-band (like 2.4GHz & 868/916 MHz) systems (note: mLRS supports dual band since a while).

What it is/is not good for should be in the description to aid people deciding how to implement, ideally with a "why this is so".
We should probably create a "microservice" doc for this and the other radio messages that captures the discussion more broadly, and link to that.

I'll shut up now and let the experts speak.

[olliw42]

The question of importance (for all fields) probably depends on who you think is the consumer.

yeah. That's why I said that these message do have, and (always) will have limitations. You always will find some system which or someone who wants one specific thing to be different. My approach thus was to see what could be a common sense minmal set of fields for the "high-frequency" message. And in fact not everything needs to be included.

In some sense, the "high-frequency" message copes just with the facts of the progress in the last years, that today not only rssi and noise is of interrest but also LQ (note, rssi and noise was already in the old message!), and that diversity is more often a thing.

The "low-frequency" message IMHO can be much more easily improved with time. Here it doesn't matter much how big it grows.

Upshot, the rates probably should be expected to be tunable depending on the channel and purpose.

The rate for the "high frequency" message is essentially given by the receiver. The canonical case will be that it's the rate of the ota frames of that link. So, canonically, it will be the same rate with which the RADIO_RC_CHANNELS message will be emitted. In essence: it's a receiver thing and it shall do whatever it thinks is appropriate. Nothing for us (MAVLink) to worry about.

The rate for the "low-frequency" message may be subject to more discussion. I would however also leave it to the receiver to decide what's reasonable. My approach here is simple, send at low rate (e.g. 0.2 Hz) but if a change in the fields happen don't wait but send it out once immediately.

That's what the streaming rate setters are for, but perhaps these need to be default values for each radio type.

I don't think the stream rate setters should be the canonical approach. First, the receiver is normally not emitting a heartbeat, which it should (IMHO) if the "normal" MAVLink rules for a component are used, e.g., if the receiver wants to have AMVLink parameters, it shoudl send the heartbeat and provide the parameters service. Similarly I would argue if the receiver wants to support the stream rate setters it should send a heartbeat and support the stream rate microservice. Second, as argued I think that in nearly all cases there is absolutely no need since it's kind of clear to the receiver what's the most mneanigfull to do.

The other way around: a component is allowed to stream without the stream setters thing.

What it is/is not good for should be in the description to aid people deciding how to implement, ideally with a "why this is so".
We should probably create a "microservice" doc for this and the other radio messages that captures the discussion more broadly, and link to that.

I'm afraid a description which would cover all ups and downs including all explanations for why would be massive. At some point some docs with guidance should seriously considered.

[JyeSmith]

Hey, I got a ping so Ill give my 2c.

I really like what @olliw42 is doing here and with RADIO_RC_CHANNELS. Previously link stats have been hacked in for individual use. So this will make life much easier.

My only useful comment is to change rx_receive_antenna/rx_transmit_antenna/tx_receive_antenna/tx_transmit_antenna to carry information for both antennas. ELRS Gemini and dual band setups can tx and rx simultaneously. It will be useful to see if both antennas are transmitting/receiving. For example if you are using a dual band system and the rx_receive_antenna says it is always on the 900M antenna. Do you know if the other band is actually receiving.

Possibly make them bit masks with 0..7 bits representing antennas 1..8?

Just a thought but I not set on the idea.

[olliw42]

@JyeSmith
that's a very good comment, we may want to spend some more thoughts though

I would suggest to not introduce more than two antenna at this point. Extension to that would raise more questions.

for the transmit fields, dualband/gemini actually means that both antenna are always transmitting, so it essentially becomes a static info, or in fact redundant info since it should be known from what hardware/config is used. In some way one starts to encode info on the hardware/config into this message. I mean, if the encoding would be with flags, 0 could be understood to mean single antenna, 1 and 2 diversity, and 3 dualband/gemini.

While working on this message the last dozen months I pondered quite a bit about such schemes, and flipped back and forth, but eventually concluded that it's best to not attempt to have any hardware/config info, as it just tends to make things more confusing. If ever needed at all it would be better placed in the other message, e.g. via a capabilties flag.

My first impuls would rather be to change specification to 0 means antenna1 or both antenna1&antenna2. To not have a static 3. One could change to 0: assume antenna1 or both antenna1&antenna2, 1: antenna1, 2: antenna2, 3: antenna1 & antenna2. Could be extended to more antenna anytime later, and would allow to keep 0 even for dualband/gemini, but is more complex.

(side note: one may notice that I've tried to arrange these fields such that the more typical the cases the more 0's at the end)

for the receive fields it's actually not only a dualband/gemini thing but also with full diversity. The current spec is rather to mean "selected antenna". Your suggestion would imply a change in meaning to actually receiving antennas. Not saying I'm against it, just noting. To avoid yet another enum we could specify 0: unknown, assume antenna1, 1: antenna1, 2: antenna2, 3: antenna1 & antenna2. (could be extended to more antenna as you suggested anytime later)

However, again, this info is in fact redundant. It can be derived from the rssi1 and rssi2 values (which makes me note that the spec I have written up so far isn't precise but misleading). If both antenna are receiving, both would not be 254 or 255, which would signal that both antenna are receiving (side note: in some way, the current spec can give you actually more info as it tells you also what antenna is selected). However, admittedly, it is more complex to interpret than the 0,1,2,3.

so, to sum up, the current message provides the info already, It's a discussion of how it's more conveniently be presented/encoded.

thoughts?

[JyeSmith]

thoughts?

Thats fine. Just throwing ideas out there for now and thinking about the future. We want to avoid coming back and tacking on more stats later.

rx_LQ_rc and rx_LQ_ser, are you seeing a difference here? Is mLRS sending them with different mod params? I would have expected these 2 LQs to be the same.

LQ_ant1 and LQ_ant2 may be better, particularly with dual band systems.

[olliw42]

yes, for mLRS rx_LQ_rc and rx_LQ_ser can be substantially different, with the rule rx_LQ_ser <= rx_LQ_rc. In general, it really depends on how the link is constructed, it can be of all kinds, equal, <=, >=, varying.

separating LQ into LQ_ant1 and LQ_ant2 doesn't make much sense to me, also not for dualband.

we kind of face the general problem that the more complex the link is the more data one can create. One thus needs to strike a balance, to avoid inflation, which sure is subjective.

@jlpoltrack e.g. has made the suggestion to add the frequency, which I like a lot since I think it can give really usefull plots allowing useful analysis, but I'm not convinced it should be added to this messge here. MAVLink and ArduPilot's lua scripting allows for a lot of flexibility.

[JyeSmith]

yes, for mLRS rx_LQ_rc and rx_LQ_ser can be substantially different, with the rule rx_LQ_ser <= rx_LQ_rc. In general, it really depends on how the link is constructed, it can be of all kinds, equal, <=, >=, varying.

separating LQ into LQ_ant1 and LQ_ant2 doesn't make much sense to me, also not for dualband.

I think we will differ here. Separate band LQs will be more useful for user who fly in noisy environments. They main request for DB I see is from Cinelifter working on sets.

rx_LQ_ser can also be obtained from the data rate.

@jlpoltrack e.g. has made the suggestion to add the frequency, which I like a lot since I think it can give really usefull plots allowing useful analysis, but I'm not convinced it should be added to this messge here. MAVLink and ArduPilot's lua scripting allows for a lot of flexibility.

OH YES!!!

But not a rate or hard coded enum, a free text section that describes the RF mode being used. The current rf mode number used with the crsf linkstats is crap. Something human readable that can be shown in the OSD. Making it free text means we can use it for any other new and weird stuff we dream up in the future. Including it in this packet isnt a bad idea because it will be captured as the link changes, for what ever reason. This is not an OTA packet, so BW isnt a problem.

[jlpoltrack]

@olliw42 @JyeSmith
Just a few notes to keep the conversation going, please correct any misunderstandings.

• mLRS uplink packet (ground to vehicle) contains both rc and serial data. There are two CRCs, one for the first section containing rc data and another for the entire packet. So while the entire packet could miss the CRC check, the first section may pass and that is what is measured by 'rx_LQ_rc' (the entire packet is measued by 'rx_LQ_ser').
• I believe the mavlink branch of ELRS will only send serial or rc within a packet and therefore the two LQs proposed don't 'fit' the current scheme.
• One suggestion would be to have two generic LQ fields. mLRS could use it one way and ELRS another.
• The idea for frequency was to expose this on each packet in the 'RADIO_LINK_STATS' message with a unsigned 32 bit integer to understand the channel in-use.
• Something like what you're describing may better fit in the 'RADIO_LINK_INFORMATION' message.

[olliw42]

@JyeSmith many thx for your thoughtfull copmments. I think we will not agree on all aspects.

First and probably most importantly of all:

This is not an OTA packet, so BW isnt a problem.

We should not pay no attention here.

Let's assume a max reasonable packet rate of 100 Hz (more than that doesn't reasonably make much sense with a MAVLink receiver). The RADIO_LINK_STATUS message as is has 30 bytes (no signature assumed), so that corresponds to 3000 bytes/sec or essentially all of a 38400 baud serial. Take into account the RADIO_RC_CHANNELS messages and we are exceeding a 57600 baud serial. One may argue now that 100 Hz is too much, but we defenitely should want to allow for 50 Hz. Secondly, sending that info in the message only makes sense if it is somehow used in the flight controller, mainly logged. Logging this much data will take a toll (e.g. gaps in the log).

So, just arbitrarily inflating the message is IMHO absolutely not the way to go. E.g. a plain text mode field in the "high frequency" message is IMHO a no go.

I think we will differ here. Separate band LQs will be more useful for user who fly in noisy environments.
rx_LQ_ser can also be obtained from the data rate.

There seem to be some logical flaws in the argument here.

The suggested separate band/antenna LQs, which if I understand correctly reflect how many ota packets are received on each antenna for a given time unit (moving averaged, filtered, or time slotted doesn't matter) can be reconstructed from the rssi field any time. So, it's redundant data. If it is determiend that this is useful, any implementation can provide that info, but it doesn't have to be send.

The rx_LQ_rc and rx_LQ_ser cannot be reconstructed in contrast (except in generic "simple" cases which we can't assume in general). Especially rx_LQ_ser cannot be reconstructed from the data rate, since the data rate is not a constant at all. Also the data rate isn't provided.

The tx_LQ_ser could in fact be reconstructed any time from the rssi, like the suggested separate band/antenna LQs. So, this single field is indeed redundant and add for convenience and symmetry.

They main request for DB I see is from Cinelifter working on sets.

I think you guys are too much biased towards the fpv world :) :D The ArduPilot/PX4 world is much bigger than that. Actually, AFAIK, MAVLink isn't really a topic at all for Cinelifters, so it's rather orthogonal in fact

The current rf mode number used with the crsf linkstats is crap.

the proposed RADIO_LINK_INFORMATION message has a field to identify the link brand and an unspecified field to encode the mode. So, any link brand can specify whatever they "want" here. Your concern has already been taken into account. It can be discussed if a plain text mode field is valuable enough to be added.

[olliw42]

@jlpoltrack

• 1st bullet: yes. But that's mLRS specifics which should not be relevant. The point is that with rc and ser data one needs two LQs since they are in general not linked or correlated.
• 2nd bullet: IMHO they absolutely make sense. Simply count how many rc's and how many ser's per time. This transmission scheme would actually be also my example to "proof" that rx_LQ_rc and rx_LQ_ser are not related to each other in general, hence the two are needed.
• 3rd bullet: We can perfectly well specify what they should mean, so better to do it.
• 4th bullet: got that. But you don't have to send the frequencies in the same message, sending two (or more) messages in sequence is fully sufficient to reconstruct and e.g. create proper plots.

[JyeSmith]

* One suggestion would be to have two generic LQ fields.  mLRS could use it one way and ELRS another.

This is a good idea.

Being a LINK STATS packet the information contained should be about the rf link, and not the data carried. It make s a lot of sense to see the LQ of a degrading band.

[IamPete1]

I realize I'm a late to the party here. But how about adding a flag for local vs remote and index for antenna number. That would mean you can drop the separate tx and tx fields and the separate antenna fields. So rather than sending one instance of the current message, for a dual antenna system you would send four messages, one per antenna on each side. This reduces the amount of data for single antenna systems and means we would support future systems with more than 2 antennas.

[olliw42]

I thought about such a thing in fact, but abandoned it for data rate waste. Note that a mavlink v2 is 12 bytes header&footer (no signature) so 4 messages of few bytes is costly compared to one message with 4xfew bytes. Also, at some point one may argue a timestamp is needed for syncronization, another 4 bytes per message. IMHO it could make sense however for supporting "unusual" cases.

given the diverse nature of philosopies, I'm currently also rather thinking that one should/could use AP lua instead.

as regards this message here, I think the use cases discussed in the above are all in fact realized in the currently proposed message. I have not yet heard any convincing argument indicating otherwise.
PS: in the message one could replace the 4 anteenna bytes by one antenna flag byte, saving 3 bytes. The message as is isn't really very wastefull.

[wvarty]

ELRS are close to moving our mavlink support out of beta-development, and into mainline in the next release.
@IamPete1 has suggested we take a look at using the new RADIO_RC_CHANNELS message, added here ArduPilot/ardupilot#26356 for our RC control, rather than the existing RC_CHANNELS_OVERRIDE message, which sounds good to me.

The one thing we are still missing though is link stats. This seems to solve that.
I cant see any strong objections to the current shape of this message in the comments above.... just some healthy dev discussion. Is there anything holding this back from progressing to a formal dev review and merge?

[FrauBluher]

Was looking into proposing a new message alternative to RADIO_STATUS and was pleased to find that others are already making progress towards this.

My constructive feedback:

1. I'm not a huge fan of TX and RX "side" fields in a single message, (TX Receive Strength is confused) and would prefer that the message contain one set of RSSIs. The RADIO_LINK_TYPE (as-is) and/or different component id should be enough to separate out which part of the link the RADIO_LINK status message is coming from. Further, there are many network topologies currently employed that can use this message (mesh, hub and spoke, point-to-point, pure broadcast, etc), for most of these having tx and rx RSSI fields in the same message doesn't really make sense for the overall link. Technically a message with both local and remote RSSI in a single message for a point-to-point solution is already embodied in RADIO_STATUS.

2. All of these mixed metrics of radio performance are always captured over some period of time. Toward the discussion of radio bandwidth utilization, capturing capture period for those metrics can aid in consumers interpreting the message correctly / supporting high / low update rates.

3. Would appreciate at least 4 antenna fields - perhaps each mixed metric can be an array of 4-8 values. As proposed I'm already beyond the message's ability to only track 2 antennas.

The basic MVP metrics that we use to take RF link configuration actions are:

• Per-antenna "RSSI"
• RSSI measurement period in uS
• Per-antenna minimum RSSI over measurement period
• Per-antenna maximum RSSI over measurement period
• Per-antenna "SNR"
• Per-antenna maximum SNR over measurement period
• Per-antenna minimum SNR over measurement period

As for the network data flow fields, I would also advocate for a set of tx_buf_outstanding and rx_buf_outstanding fields to track issues with output bursts, input receive handling, etc.

[hamishwillee]

@olliw42 et al, Any comment to the last post? ^^^

[olliw42]

ok, it seems I should spend time for some response. You'll get my opinion, if it's liked or not. :)

I think this is one of the topics where there are more opinions than people with opinions. The few comments so far IMHO have made it clear that it is fundamentally not possible to draft one message which makes everyone, i.e. all varieties of existing or future links with their respective needs and desires, happy ... unless the message is essentially an opaque container, as it also was suggested. Just way too divergent requirements. At this point I would argue that yet another opaque mavlink message is not needed, given there are already several, and not desirable, given that MAVLink - at least that's my understanding - tries to rather standardize than to opaquize.

I could make an proposal, since for my entertainement I thought about possible ways and came to some conclusion, but I don't felt I would spend my time well with suggesting it. It would need compromises, which I don't see every side is prepared for, would need finding consensus on what a reasonable minimal common sense could be, and possibly also would need changes in practice with regards to MAVLink on ArduPilot side which in the past they appeared to have been pretty stiff on. That's why I didn't do, and still feel it's pointless to do.

As regards your original question, comments on the last post: All reasonable points, from the respective point of view, but it's yet another brick in the wall I just described, and not making things easier. Just adding more and more requirements and desires is obviously not a path to success.

You know, in German we have the saying: Zuviele Köche verderben den Brei.

Not very constructive, I know, but I think it has a "hidden" lesson as regards to what I would think would have to be done. :)

[hamishwillee]

@olliw42 Interestingly, we had a similar conversation at the MAV call last night, driven by @auturgy .

@FrauBluher "As I understand it", @auturgy made the point that the use case you seem interested in addressing is perhaps not the same as most of the stakeholders in this call. I think you want a lot of high rate information about the network in order to provide information needed for managing a complex network. The main audience for the messages under discussion is a relatively low rate message to give a high level overview of potentially problematic links.

Other points like having separate messages are my preference too, but some flash can be saved by having a single message.

If I'm correct about your use case then I think that is a separate set of messages, possibly ones that should be dialect specific or custom message. The reasoning being that this kind of detail is not what most users want most of the time, or would be willing to pay the firmware/channel costs of implementing.

Hope that makes sense. @olliw42 what is blocking this PR otherwise?

[FrauBluher]

If folks don't want to wait to add this new message, please go ahead with my blessing - though I'd like to make it known that this new message has issues that will prevent it from being used alone in our mavlink speaking radios. Before I make that leap I will propose yet another new message to see if folks are willing to accept the hit to flash. (Is this still really a major concern? What is the overall footprint currently with the latest message set?)

[hamishwillee]

Flash is just "generally a concern" - enough that some flight stacks comment out certain messages and work towards removing any that are not essential. So it is not a concern to me personally, but I know that sometimes things will be blocked.

That said, you are more than welcome to outline their form here - ease of use can sometimes be a justification for inclusion,.

[olliw42]

@olliw42 what is blocking this PR otherwise?

the RADIO_LINK_STATS message has not received much agreement here but just suggestions for how to modify ...

from my side there is really not much blocking it. I had changed it slightly; the last four bytes can be put into one flag, saving 3 bytes, also making it a bit more clear, and I added two frequency float field which can be set to 0 if not used (= then no cost). So, that's what I would suggest over what you see here. (and I obviously believe it strikes a good compromise LOL) If such message appears acceptable for being added, I can update the PR (in 1-2 weeks, just leaving for vaccation)

My proposal btw would be based on the realization that lua could be used for anything which is special, like the above. MAVLink and ArduPilot lua provide all infrastructure to do that, it may need few adaptions here and there, and if ArduPilot would adapt their practices as regards to e.g. dialects this should be very functional and allow everyone doing what they need.