Homebrew, open source, repurposed, hacked, software defined, open hardware

Monday, 7 September 2015

Codec2 700B mode trellis decoding experiments and Reverend Thomas Bayes

The codec2 700B mode has been an improvement on the 700 mode, and employs vector quantisation (VQ) rather than scalar quantisation.

The characteristics of the 700B frames can be summarised as follows:

  bits_per_frame = 28;       % number of bits/frame for "700B" mode
  bit_fields = [1 5 3 6 6 6 1]; % number of bits in each field
  field_labels = ["voicing"; "logWo"; "energy"; "LSPmelVQ1"; "LSPmelVQ2"; "LSPmelVQ3"; "spare"];

which contrasts to the features of the 700 mode:

  bits_per_frame = 28;
  bit_fields = [1 5 3 3 2 4 3 3 2 2];
  field_labels = ["voicing"; "logWo"; "energy"; "LSP1"; "LSP2"; "LSP3"; "LSP4"; "LSP5"; "LSP6"; "spare"];

Naturally, trellis decoding seemed worth trying on 700B.

Here are the transition probability plots for the 700B bit fields, when the ve9qrp10s sample is processed.





It is interesting to note the quite striking difference in appearance between the very uniformly dispersed nature of the 700B VQ bit field transition probability mesh plots and the "peakier" 700 scalar LSP bit field mesh plots with more marked central tendencies ( see: codec2-700-mode-trellis-decoding )

This suggests that the VQ is doing a pretty good job of encoding information without much redundancy, which is likely to have implications for maximum likelihood decoding strategies. It is harder to derive a useful measure of the central tendency and then meaningfully apply it when the mesh plot looks like a square of uniformly cut lawn, as opposed to a nice mound in the middle of the lawn.

codec2's author, David Rowe, also discusses this issue of information redundancy in his blog.

It seems to be all about that hoary old chestnut that was first discussed in 1763 by the Reverend Thomas Bayes in "An Essay towards solving a Problem in the Doctrine of Chances". The posterior odds, i.e. our estimation of the most likely received codeword, can only be improved if we have useful information regarding the prior distribution of likely codewords. In the absence of useful information, we are left with a non-informative entropy maximizing distribution.



Here are some summary statistics from the first decoding runs:

Passing through bitfield 4 LSPmelVQ1 without trellis decoding
Passing through bitfield 5 LSPmelVQ2 without trellis decoding
Passing through bitfield 6 LSPmelVQ3 without trellis decoding
Passing through bitfield 7 spare     without trellis decoding
processing 700B mode parameter: 1, nbits: 1, label: voicing                         
Eb/No: 0.00 dB nerrors 25 13 BER: 0.10 0.05 std dev: 0.31 0.23
processing 700B mode parameter: 2, nbits: 5, label: logWo                           
Eb/No: 0.00 dB nerrors 121 82 BER: 0.10 0.07 std dev: 4.71 4.67
processing 700B mode parameter: 3, nbits: 3, label: energy                          
Eb/No: 0.00 dB nerrors 67 63 BER: 0.09 0.08 std dev: 1.20 1.32

And an attempt to trellis decode the VQ encoded bitfields (four hours of octave on a dedicated 64 bit 2.7GHz intel cpu):

Passing through bitfield 7 spare     without trellis decoding
processing 700B mode parameter: 1, nbits: 1, label: voicing                         
Eb/No: 0.00 dB nerrors 25 13 BER: 0.10 0.05 std dev: 0.31 0.23
processing 700B mode parameter: 2, nbits: 5, label: logWo                           
Eb/No: 0.00 dB nerrors 121 82 BER: 0.10 0.07 std dev: 4.71 4.67
processing 700B mode parameter: 3, nbits: 3, label: energy                          
Eb/No: 0.00 dB nerrors 67 63 BER: 0.09 0.08 std dev: 1.20 1.32
processing 700B mode parameter: 4, nbits: 6, label: LSPmelVQ1                       
Eb/No: 0.00 dB nerrors 246 115 BER: 0.17 0.08 std dev: 15.57 10.97
processing 700B mode parameter: 5, nbits: 6, label: LSPmelVQ2                       
Eb/No: 0.00 dB nerrors 277 115 BER: 0.19 0.08 std dev: 15.42 10.80
processing 700B mode parameter: 6, nbits: 6, label: LSPmelVQ3
Eb/No: 0.00 dB nerrors 285 114 BER: 0.19 0.08 std dev: 14.55 10.52

Judging by the large increases in errors from our attempt to apply maximum likelihood decoding to the VQ bit fields, it seems reasonable to conclude that the VQ encoded bit fields are doing an excellent job of conveying a lot of information with minimal redundancy. Unfortunately for us, this would suggest that we can't profitably employ maximum likelihood decoding for the VQ bit fields directly.

While experimenting, we have also seen that Reverend Bayes' insights are quite relevant to low bit rate audio codec R&D some 250 years later!

Here are some .wav files of the trellis decoded ve9qrp_10s sample which has had additive gaussian white noise added after 700B encoding, in keeping with the method used for the other codec2 mode trellis decoding experiments....

No bit fields trellis decoded:
ve9qrp_10s_700B_0.50_simple_700B.wav

Bit field 1 trellis decoded:

ve9qrp_10s_700B_0.50_trellis_700B_dec1.wav

Bit field 2 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec2.wav

Bit field 3 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec3.wav

Bit fields 1,2 trellis decoded:

ve9qrp_10s_700B_0.50_trellis_700B_dec12.wav

Bit fields 1,3 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec13.wav


Bit fields 2,3 trellis decoded:

ve9qrp_10s_700B_0.50_trellis_700B_dec23.wav


Bit fields 1,2,3 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec123.wav


Bit fields 1,2,3,4,5,6 trellis decoded
ve9qrp_10s_700B_0.50_trellis_700B_dec123456.wav


Bit field 4 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec4.wav

Bit field 5 trellis decoded:

ve9qrp_10s_700B_0.50_trellis_700B_dec5.wav


Bit field 6 trellis decoded:
ve9qrp_10s_700B_0.50_trellis_700B_dec6.wav


The intelligibility of the samples with bit fields 4, 5, or 6 decoded are not improved, as would be expected based on the very uniform distribution of the encoded VQ values evident in the transition probability plots, and also from the significantly increased number of bit errors and  standard deviation seen in the summary statistics.

From the above samples, it is clear that bit fields 1,2 and 3, either singly or in combination, benefit from direct trellis decoding, but the VQ bit fields would require decoding before attempting trellis decoding of their encoded information.


Codec2 700 mode trellis decoding experiments

Further to the codec2 1600 bit/second mode experiments with trellis decoding, the octave script has been modified further with a view to allowing any mode of interest to be specified.

As part of this refactoring and testing of the modified octave script, the 700 mode has been put through the script, with a combination of trellis decoded and passed through bit fields.

Here are the transition probability mesh plots for all of the bit fields showing the likelihoods (in Z axis height) of a given bitfield transitioning from a given value (X-axis) to a corresponding value on the Y axis. These plots were generated with the ve9qrp10s sample which was also used for the 1600 mode trellis decoding experiments.

The voicing bit field appears to be the least impressive in terms of predictability, and this bitfield may not lend itself to trellis decoding without excess errors being introduced.











Here are the summary statistics for the bitfields following the addition of additive Gaussian white noise (AGWN) and then trellis decoding.

Passing through bitfield 10 spare   without trellis decoding
processing 700A mode parameter: 1, nbits: 1, label: voicing                       
Eb/No: 0.00 dB nerrors 23 17 BER: 0.09 0.07 std dev: 0.30 0.26
processing 700A mode parameter: 2, nbits: 5, label: logWo                         
Eb/No: 0.00 dB nerrors 121 85 BER: 0.10 0.07 std dev: 4.95 4.83
processing 700A mode parameter: 3, nbits: 3, label: energy                        
Eb/No: 0.00 dB nerrors 63 65 BER: 0.08 0.09 std dev: 1.28 1.31
processing 700A mode parameter: 4, nbits: 3, label: LSP1                          
Eb/No: 0.00 dB nerrors 64 55 BER: 0.09 0.07 std dev: 1.06 1.29
processing 700A mode parameter: 5, nbits: 2, label: LSP2                          
Eb/No: 0.00 dB nerrors 47 50 BER: 0.09 0.10 std dev: 0.63 0.71
processing 700A mode parameter: 6, nbits: 4, label: LSP3                          
Eb/No: 0.00 dB nerrors 63 66 BER: 0.06 0.07 std dev: 1.83 2.41
processing 700A mode parameter: 7, nbits: 3, label: LSP4                          
Eb/No: 0.00 dB nerrors 78 61 BER: 0.10 0.08 std dev: 1.30 1.25
processing 700A mode parameter: 8, nbits: 3, label: LSP5                          
Eb/No: 0.00 dB nerrors 42 64 BER: 0.06 0.09 std dev: 0.87 1.31
processing 700A mode parameter: 9, nbits: 2, label: LSP6                          
Eb/No: 0.00 dB nerrors 32 42 BER: 0.06 0.08 std dev: 0.49 0.66

And another run:

Passing through bitfield 1 voicing without trellis decoding
Passing through bitfield 10 spare   without trellis decoding
processing 700A mode parameter: 2, nbits: 5, label: logWo                         
Eb/No: 0.00 dB nerrors 140 103 BER: 0.11 0.08 std dev: 5.20 5.28
processing 700A mode parameter: 3, nbits: 3, label: energy                        
Eb/No: 0.00 dB nerrors 60 55 BER: 0.08 0.07 std dev: 1.08 1.21
processing 700A mode parameter: 4, nbits: 3, label: LSP1                          
Eb/No: 0.00 dB nerrors 70 61 BER: 0.09 0.08 std dev: 1.21 1.36
processing 700A mode parameter: 5, nbits: 2, label: LSP2                          
Eb/No: 0.00 dB nerrors 34 33 BER: 0.07 0.07 std dev: 0.53 0.53
processing 700A mode parameter: 6, nbits: 4, label: LSP3                          
Eb/No: 0.00 dB nerrors 72 71 BER: 0.07 0.07 std dev: 1.85 2.63
processing 700A mode parameter: 7, nbits: 3, label: LSP4                          
Eb/No: 0.00 dB nerrors 58 50 BER: 0.08 0.07 std dev: 1.23 1.17
processing 700A mode parameter: 8, nbits: 3, label: LSP5                          
Eb/No: 0.00 dB nerrors 42 63 BER: 0.06 0.08 std dev: 0.85 1.21
processing 700A mode parameter: 9, nbits: 2, label: LSP6                          
Eb/No: 0.00 dB nerrors 34 37 BER: 0.07 0.07 std dev: 0.57 0.59

And another run:

Passing through bitfield 1 voicing without trellis decoding
Passing through bitfield 2 logWo   without trellis decoding
Passing through bitfield 10 spare   without trellis decoding
processing 700A mode parameter: 3, nbits: 3, label: energy                        
Eb/No: 0.00 dB nerrors 44 53 BER: 0.06 0.07 std dev: 1.07 1.15
processing 700A mode parameter: 4, nbits: 3, label: LSP1                          
Eb/No: 0.00 dB nerrors 80 65 BER: 0.11 0.09 std dev: 1.17 1.31
processing 700A mode parameter: 5, nbits: 2, label: LSP2                          
Eb/No: 0.00 dB nerrors 32 39 BER: 0.06 0.08 std dev: 0.53 0.62
processing 700A mode parameter: 6, nbits: 4, label: LSP3                          
Eb/No: 0.00 dB nerrors 69 70 BER: 0.07 0.07 std dev: 1.68 2.51
processing 700A mode parameter: 7, nbits: 3, label: LSP4                          
Eb/No: 0.00 dB nerrors 69 61 BER: 0.09 0.08 std dev: 1.27 1.25
processing 700A mode parameter: 8, nbits: 3, label: LSP5                          
Eb/No: 0.00 dB nerrors 38 65 BER: 0.05 0.09 std dev: 0.85 1.27
processing 700A mode parameter: 9, nbits: 2, label: LSP6                          
Eb/No: 0.00 dB nerrors 27 35 BER: 0.05 0.07 std dev: 0.49 0.58

After all of that, here is what the ve9qrp_10s_700A.bit sample sounds like with simple decoding after the addition of the AWGN:

ve9qrp_10s_700A_0.50_simple_700A.wav

Here is what the ve9qrp_10s_700A.bit sample sounds like with trellis decoding of all bit fields except for the  spare bit field (10) after the addition of the AWGN:

ve9qrp_10s_700A_0.50_trellis_700A_dec123456789.wav

Here is what the ve9qrp_10s_700A.bit sample sounds like with trellis decoding of bit fields 2-9, skipping the voicing (1) and the spare (10) bit fields after the addition of the AWGN:

ve9qrp_10s_700A_0.50_trellis_700A_dec23456789.wav

Here is what the ve9qrp_10s_700A.bit sample sounds like with trellis decoding of bit fields 3-9, skipping the voicing (1), logW0 (2) and the spare (10) bit fields after the addition of the AWGN:

ve9qrp_10s_700A_0.50_trellis_700A_dec3456789.wav

Having confirmed that the modified script still works with the codec2 700 mode, the next step is to support the 700B and 1300 bit/second modes.

In closing, all of this experimentation serves to highlight that black box vocoders subject to intellectual property protection being marketed by commercial equipment vendors do not allow this sort of experimentation by interested amateurs.

Sunday, 6 September 2015

gEDA PCB footprint text utility

When creating footprints in gEDA, the addition text elements other than the overlaid refdes is not supported.

Here is a command line java utility that allows text to be generated as an element which can then be loaded and added to a gEDA PCB footprint or layout.

https://github.com/erichVK5/FootprintTextForPCB

Without a utility of this nature, those making footprints and seeking to add text have to hand draw the desired text.

v1.1 of the utility uses the free Hershey Sans 1 Stroke Font and outputs 0.01mil (imperial, square bracketed) units.

This builds on the previous font work discussed previously:

geda-pcb-font-utilities-and-options

I plan to incorporate this utility into the Kicad to PCB footprint importer utility,

https://github.com/erichVK5/KicadModuleToGEDA

to allow text defined in Kicad modules to be rendered when converting to gEDA PCB format.

I may also add support for other ported fonts in due course.

Hints for usage:

Step 1)

decide on the text (or texts) needed in the footprint being designed

Step 2)

generate each needed bit of text as a footprint with this utility, i.e.
 
java FootprintTextForPCB -t "You Shouldn't Have Unsoldered This" -m 1.3

which will generate a footprint file
 
You_Shouldn_t_Have_Unsoldered_This.fp

that will contain the text. It will recognise spaces and any of the usual ASCII characters that PCB can ordinarily display as text, but, you will need to escape characters that the shell might take exception to, and the escape character may end up getting rendered in the footprint text, until such time as I support escape characters a bit better.

Step 3)

When generating the new footprint in PCB, use

"File:Load Element To Buffer"

to load the newly generated footprint onto the layout.
Place the text in a suitable position. If it is the wrong size, go back and play with the magnification ratio option.

Step 4)

Select the text by clicking on it.

CTRL-x to cut the text to buffer

Go to "Buffer:Break Element To Pieces" to convert the text footprint into silk line primitives, and click to place the broken up element where it is needed.

Hit "Esc" to deselect.

Step 5)

Proceed now, as you normally would, to convert your collection of elements (which now include the silk lines showing the text) making up your footprint in its entirety into a footprint.

Usage:
 
java FootprintTextForPCB -t "my Text For Conversion To Silkscreen Stroke Elements" -m X.XXXX

"my Text For Conversion To Silkscreen Stroke Elements" is ASCII text, which can include spaces,
and X.XXXX is an optional magnification ratio; default = 1.0)

If run without any command line arguments, a demonstration footprint file
called demonstration1234567890.fp, will be generated

Installation:

1) install a java compiler and java virtual machine (JVM) using your preferred package management system/source, if it isn't already installed.

2) clone the FootprintTextForPCB git repository. Failing that, download the java source file and put it in a suitable directory with the same subdirectories and contents.

3) in the FootprintTextForPCB directory, type:

user@box:~$ javac FootprintTextForPCB.java

and that should be it, you are now ready to use the FootprintTextForPCB utility.

Codec2 1600 bit/s mode trellis decoding experiments

Codec2 is an open source low bit rate voice coder (vocoder) that enables voice to be carried on data channels at very low data rates.

Low bit rate vocoders are distinct from their higher bit rate encoder cousins such as mp3 which seek to reproduce more than just voice, i.e. music.

Codec2 is particularly exciting owing to its potential to revolutionise HF voice communications, until now dominated by Single Sideband Transmission (SSB). Codec2 already offers better robustness than SSB in low signal to noise conditions. Codith ec2 also has significant potential in VHF and above amateur radio communications, where single frequency time division multiple access technologies (TDMA) already in use commercially for mobile communications and telephony can be introduced with significant spectrum saving benefits in amateur bands and on amateur repeaters.

Anyway, after playing with trellis.m in the codec2-dev/octave directory, support for trellis decoding of the 1600 bit/s mode was implemented.

Maximum likelihood decoding is used, using the raw hts sample audio file encoded at 1600 bit/s as the source of the training database.

David Rowe has pointed out that that 1600 bit/s mode uses an underlying 1300 bit/s bit stream plus a 300 bit/s forward error correction (FEC) bit stream, for a total of 1600 bits per second.

Accordingly, experiments on the underlying 1300 bit/s mode is planned, in the absence of FEC.

Additive white gaussian noise has been added to the ve9qrp_10s audio sample, followed by trellis decoding of various combinations of the bit fields.

This is the result of decoding after the addition of noise to the codec2 bitstream, with no trellis decoding:

ve9qrp_10s_1600_0.50_simple_1600.wav

This is the result of decoding after the addition of noise to the codec2 bitstream, with trellis decoding of bitfields 3,6,7,8,9,10,11,12,14,15,16:

ve9qrp_10s_1600_0.50_trellis_1600_dec3678910111213141516.wav

This is the result of decoding after the addition of noise to the codec2 bitstream, with trellis decoding of bitfields 1,3,4,6,7,8,9,10,11,12,14,15,16:

ve9qrp_10s_1600_0.50_trellis_1600_dec134678910111213141516.wav

This is the result of decoding after the addition of noise to the codec2 bitstream, with trellis decoding of bitfields 7,8,9,10,11,12,14,15,16:

ve9qrp_10s_1600_0.50_trellis_1600_dec78910111213141516.wav

This is the result of decoding after the addition of noise to the codec2 bitstream, with trellis decoding of bitfields 1,4,7,8,9,10,11,12,14,15,16:

ve9qrp_10s_1600_0.50_trellis_1600_dec1478910111213141516.wav

1600 mode bitfields are as follows:

 bits_per_frame = 64;
 bit_fields = [2 7 5 2 7 5 4 4 4 4 4 4 4 3 3 2];
 field_labels = ["voicing1"; "scalarWo1"; "energy1"; "voicing2"; "scalarWo2"; "energy2"; "LSP1"; "LSP2"; "LSP3"; "LSP4"; "LSP5"; "LSP6"; "LSP7"; "LSP8"; "LSP9"; "LSP10"];

Intelligibility in the presence of noise seems to be enhanced the most by maximum likelihood decoding of the LSPs and voicing bits, with next best being maximum likelihood decoding of just the LSPs.

Maximum likelihood decoding of just the LSPs in the 1600 bit/s mode is not too demanding given the small bitfield lengths. Based on this admittedly limited sample set, maximum likelihood decoding of the voicing bits in addition to the LSPs seems to reduce the occasional "stutter" type artifacts.

Errors introduced by maximum likelihood decoding of the energy bitfields seem to have an adverse effect on intelligibility. Interestingly, there were dense probabilities in the high order bits, and sparse probabilities in the low order bits... see transition probability plots below, and see the summary statistics showing significantly increased errors with trellis decoding.

Maximum likelihood decoding of the scalar W0 bitfields has not been performed owing to the bits = 7, making processing by octave quite challenging. This is due to the exponential ( i.e. 2^(bitfield length) ) demands of the maximum likelihood decoding. Further experimentation with maximum likelihood decoding of the scalar W0 bitfields in C is planned. Also of interest were the uniformly dense probabilities in the low order bits, and sparse probabilities in the high order bits... see transition probability plots below... which may have an impact on trellis decoding effectiveness - but this may reflect the effects of FEC, in which case it is actually desirable. Furthermore, the ability to adequately encode outliers is also important to convey and preserve intelligibility.

A couple of the bitfield transition probability plots also showed marked clustering into four peaks, perhaps suggesting that quantising could take better advantage of the available bits, although it may just be evidence of the FEC at work, adding robustness, or again, reflecting a required ability to adequately encode the occasional outlier.

The following plots are labeled and are presented in order of the 1600 bit/s bitfields. Basically, the Z axis shows the frequency at which a bitfield value on the X axis maps to another value on the Y axis. Accordingly, the more densely clustered the maxima, the more predictable changes from one bitfield to the next will be, and the greater the ability of the trellis decoding to make informed guesses about the most likely codeword:
















The summary statistics produced by octave:

Passing through bitfield 2 scalarWo1 without trellis decoding
Passing through bitfield 5 scalarWo2 without trellis decoding
processing parameter: 1, nbits: 2, label: voicing1 
Eb/No: 0.00 dB nerrors 34 36 BER: 0.07 0.07 std dev: 0.63 0.57
processing parameter: 3, nbits: 5, label: energy1  
Eb/No: 0.00 dB nerrors 133 98 BER: 0.11 0.08 std dev: 4.11 4.66
processing parameter: 4, nbits: 2, label: voicing2 
Eb/No: 0.00 dB nerrors 31 40 BER: 0.06 0.08 std dev: 0.61 0.65
processing parameter: 6, nbits: 5, label: energy2  
Eb/No: 0.00 dB nerrors 126 88 BER: 0.10 0.07 std dev: 4.69 4.96
processing parameter: 7, nbits: 4, label: LSP1     
Eb/No: 0.00 dB nerrors 83 72 BER: 0.08 0.07 std dev: 2.44 2.24
processing parameter: 8, nbits: 4, label: LSP2     
Eb/No: 0.00 dB nerrors 98 95 BER: 0.10 0.10 std dev: 2.46 2.55
processing parameter: 9, nbits: 4, label: LSP3     
Eb/No: 0.00 dB nerrors 104 75 BER: 0.10 0.08 std dev: 2.63 2.89
processing parameter: 10, nbits: 4, label: LSP4     
Eb/No: 0.00 dB nerrors 80 83 BER: 0.08 0.08 std dev: 2.43 2.64
processing parameter: 11, nbits: 4, label: LSP5     
Eb/No: 0.00 dB nerrors 66 75 BER: 0.07 0.08 std dev: 2.02 2.34
processing parameter: 12, nbits: 4, label: LSP6     
Eb/No: 0.00 dB nerrors 67 81 BER: 0.07 0.08 std dev: 1.54 2.82
processing parameter: 13, nbits: 4, label: LSP7     
Eb/No: 0.00 dB nerrors 82 81 BER: 0.08 0.08 std dev: 2.71 2.41
processing parameter: 14, nbits: 3, label: LSP8     
Eb/No: 0.00 dB nerrors 86 61 BER: 0.12 0.08 std dev: 1.36 1.29
processing parameter: 15, nbits: 3, label: LSP9     
Eb/No: 0.00 dB nerrors 17 50 BER: 0.02 0.07 std dev: 0.43 1.14
processing parameter: 16, nbits: 2, label: LSP10    
Eb/No: 0.00 dB nerrors 25 29 BER: 0.05 0.06 std dev: 0.31 0.49

In conclusion, early indications are that trellis decoding has the potential to improve the performance of the 1600 bit/s codec2 mode in the presence of noise, as envisaged by codec2's author, David Rowe. Further investigation is planned of the 1300 bit/s mode prior to the addition of FEC in the 1600 bit/s mode, and also the lower bit rate 700B mode that does not employ FEC. 

See also:

Rowetel blog posting "Trellis Decoding for Codec 2"

and for information on codec2 in general:

http://www.rowetel.com/blog/?page_id=452

Monday, 5 January 2015

SA GRN trunked scanning on the BC780 XLT

The South Australian (SA) Government Radio Network (GRN) uses Motorola trunked UHF radio communications for public services including Country Fire Service (CFS) control communications, SA Police Force (SAPOL), and the SA Ambulance Service (SAAS).
 
Conventional scanners provide only frustrating snippets of conversations.
 
The Uniden Bearcat BC  780 XLT is a fairly cheap trunk tracking scanner and readily obtained second hand that is compatible with the SA GRN.
 
The trunked scanning programming process is not very intuitive.
 
 
Stage 1: a worked example using bank 1 for SA GRN scanning:
 
The first thing to do is delete the frequencies in a bank you will dedicate to trunked SA GRN scanning
 
A bank on the BC780XLT is a set of 50 frequencies, i.e. stored frequencies
 
bank 1: 1-50,
bank 2: 51-100
bank 3: 101-150, etc...

in the case of bank 1:

pressing "1" on the keypad and "E" takes you to stored frequency number 1

if you then press "0" and then "E", it will delete the frequency in stored frequency number 1

if you then hit the "MAN" = manual key, it will step you to frequency number 2

if you then press "0" and then "E", it will delete the frequency in stored frequency number 2

and you can do this until you get up to and including stored frequency number 50, if you want to delete bank 1
 
Stage 2: configuring bank 1 for motorola compatible trunked scanning

the next step is to set up bank 1 (in the case of this example) as a motorola trunking bank (I have drawn on and added to the sascan instructions here):

Step 1:  Press MENU

Step 2:  Select TRUNK DATA and press E

Step 3:  Press the UP or DOWN arrows to select the trunk bank you want to program the channels into ( you can also use the VFO knob to scroll through the banks as well). When you have you bank selected (in the case of this example, we will choose bank 1), press E

Step 4:  Press the UP or DOWN arrows to select TRUNK TYPE and press E

Step 5:  Choose the Type 2 UHF trunk type. This is type no 2 and the screen should show it as TYPE 2 UHF. When this is done, press E

Step 6: Press the UP or DOWN arrows to display "BASE CONFIG 1" and press E.

Step 7:  Enter the first GRN base frequency of 412.475 and again press E

 Step 8:  The display will change so as to ask you for the spacing frequency. Enter 12.5 and press E

Step 9:  The display changes once again to ask for the offset value. Enter 380 and press E.

Step 10:  The scanner will prompt you now for the second base. Enter 415.125 and press E.

Step 11:  The scanner will now ask for the spacing and offset, the same as steps 8 and 9. Enter 12.5 for the spacing and 460 for the offset. repeater as steps 8 and 9 above.

Step 12:  Press the menu button repeatedly to exit from this mode. Then select the first channel in the bank you selected as the trunk bank by using the manual and numbers keys as you normally would.

Stage 3: find details for your local SA GRN tower

Assuming you live in the vicinity of goolwa, your local SA GRN site is the Port Elliot tower:

SA GRN Site - Pt Elliot
Location - GRN SITE BROWN HILL CROWSNEST RD PT ELLIOT 
Lat/Long  -35 30 22 / +138 38 44

Data Channel
413.125

Voice Channels
415.125
415.625
416.125
416.625
417.125
417.625

Stage 4: program the GRN channels into the bank selected for SA GRN scanning

so, you then need to program your local GRN site, in this case, we are doing it in bank 1, which starts at stored frequency number 1:

pressing "1" on the keypad and "E" takes you to stored frequency number 1

enter
 
413.125
 
then press "E" to store it, then press and hold the TRUNK key for 2 seconds. A tone will sound and the display will show the M icon

hit the manual ("MAN") key to step to frequency slot number 2

pressing "1" on the keypad and "E" takes you to stored frequency number 1

enter
 
415.125
 
then press "E" to store it, then press and hold the TRUNK key for 2 seconds. A tone will sound and the display will show the M icon just above the most significant digit of the frequency readout

hit the manual ("MAN") key to step to frequency slot number 3.... and so on

continue doing this for the remaining frequencies for the SA GRN site


Stage 5: Try some trunked scanning

having stored the Port Elliot SA GRN frequencies, the next step is to start scanning.

Press the "SCAN" button

make sure bank 1 is selected for scanning. The row of numbers 1-10 on the LCD under the word "BANK" show you which banks have been selected for scanning. In this case, make sure "1" is selected. The other banks can be excluded from the scanning.

then press the "TRUNK" button

then press the "SRCH" button

this will start the scanner reception of the data channel, and identification of trunked "conversation IDs"

You will see things something like "MOT ID: 17584" displayed on the screen below the frequency readout during overheard conversations or data bursts.

If you hear data coming through you can press the "L/O" = lockout button while the data burst is still coming through to lock out that particular trunked conversation ID. The data bursts are likely pagers or encrypted SAPOL and the scanner can be told to ignore them in this way. After listening for a while, and locking out trunked data IDs, you'll only be left with in the clear voice traffic (See also page 48 of the bearcat manual, which pretty much says the same thing).

If you have accidentally locked out some IDs by pressing the "L/O" button during voice traffic you can clear all of the locked out IDs at once (see also page 41 of the manual if you (unlikely) want to undo a lockout of a particular ID):

To unlock all locked out IDs in a bank at once:

1. Press MENU.
2. Select TRUNK DATA - BANK NO. - L/O ID REVIEW
using L, M or VFO and pressing E, SELECT or VFO.
3. Select the locked out ID you want to unlock using L or M.
4. Press E instead of L/O in step 4 above.
5. To exit from this mode, press MENU repeatedly.


The above shouldn't take more than 10-20 minutes while sitting in front of the TV.
 
If you screw something up, you just start again. 

Monday, 13 October 2014

Citrix Access Gateway 2010 phoenix BIOS and lubuntu linux installation

Hmm, an unloved, discarded, Citrix Access Gateway. IEC power socket and a USB socket at the back, as well as two ethernet ports, and a DB9 serial port at the front. Are those mini DIN keyboard and mouse sockets hiding behind the front panel?
I wonder if we can repurpose this as a rack mounted linux box? 


 Inspection of the back panel confirms it is a Citrix Access Gateway (CAG) model 2010


Time to void the warranty...


It's got a super server motherboard, 80 GB HDD, P4 pentium processor, a PCI card riser and 2x512MB = 1GB RAM on board, and it seems we don't need to do a headless server installation of linux, since it has a VGA port and a few more USB ports available.


The server booted off a lubuntu 14.04 USB installer without complaint. Luckily the BIOS did not need to be told to boot off the USB stick ahead of the HDD.


The installation went smoothly, and sucked down the additional packages over ethernet. sshd was also installed to allow access over the local network.
 

The box didn't want to power down fully, or quieten the cooling fans, which were running at full speed. The BIOS was locked down with a password, so hitting Delete on booting wasn't helpful.

Attempts to short the JBT1 pads with the power disconnected and leaving the lithium cell CMOS battery out overnight did not get rid of the BIOS password.

The python script listed below for Phoenix BIOS password checksums was found and successfully used to brute force the Citrix Access Gateway BIOS password checksum [06626].

The BIOS password error checksum was: [06626]
The Citrix Access Gateway 2010 BIOS password was found to be: bfsyxxm

The Citrix Access Gateway 2010 model uses a generic pheonix BIOS checksum, it seems.

Entering the BIOS password allowed the BIOS password to be changed to something easier to remember, and the "always power on", and "full speed" fan settings were able to be changed, to make the unit quieter and able to power down properly.

Here's the python script used to brute force the BIOS password:
#!/usr/bin/python

# Copyright 2009:  dogbert <dogber1@gmail.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

# This script generates master passwords which can be used to unlock the BIOS
# password of most Phoenix BIOS versions. It also works for some versions of
# FSI, HP and Compaq laptops which use slightly different hashing algorithms
# in the BIOS.
# You have to install python 2.x for running this script.

import os, random

keyboardDict = {  2: '1',  3: '2',  4: '3',  5: '4',  6: '5',  7: '6',  8: '7',  9: '8', 10: '9', 11: '0',
                 16: 'q', 17: 'w', 18: 'e', 19: 'r', 20: 't', 21: 'y', 22: 'u', 23: 'i', 24: 'o', 25: 'p',
                 30: 'a', 31: 's', 32: 'd', 33: 'f', 34: 'g', 35: 'h', 36: 'j', 37: 'k', 38: 'l',
                 44: 'z', 45: 'x', 46: 'c', 47: 'v', 48: 'b', 49: 'n', 50: 'm' }

def keyboardEncToAscii(inKey):
    out = ""
    for c in inKey:
        if c != 0: out += keyboardDict[c]
    return out

def asciiToKeyboardenc(inAscii):
    out = []
    asciiDict = dict([(a,k) for k,a in keyboardDict.iteritems()])
    for c in inAscii:
        if c != 0: out.append(asciiDict[c])
    return out


# The phoenix implementation of the CRC-16 contains a rather severe bug
# quartering the image space of the function: both the first and second MSB
# are always zero regardless of the input.
# For a working implementation, you'd have to change the polynom from 0x2001
# to e.g. 0xA001.
def badCRC16(pwd, salt=0):
    hash = salt
    for c in pwd:
        hash ^= c
        for i in range(0,8):
            if (hash & 1):
                hash = (hash >> 1) ^ 0x2001
            else:
                hash = (hash >> 1)
    return hash


def bruteForce(hash, salt=0, digitsOnly=False, charsOnly=True, minLen=3, maxLen=8):
    global keyboardDict
    keyboardDictOrig = keyboardDict
    if digitsOnly:
        keyboardDict = dict(zip(list(keyboardDict.keys())[0:9],list(keyboardDict.values())[0:9]))
    elif charsOnly:
        keyboardDict = dict(zip(list(keyboardDict.keys())[10:36],list(keyboardDict.values())[10:36]))

    encodedPwd = []
    for i in range(0, 7):
        encodedPwd.append(list(keyboardDict.keys())[0])
    random.seed()
    if hash > 0x3FFF:
        return "invalid hash code"
    while 1:
        # generate random password
        rndVal = random.random()*len(keyboardDict)
        for i in range(0,len(encodedPwd)):
            value = int(rndVal % len(keyboardDict))
            encodedPwd[i] = list(keyboardDict.keys())[value]
            rndVal = rndVal * len(keyboardDict)
        # test substrings of the random password
        for i in range(minLen, maxLen+1):
            if badCRC16(encodedPwd[0:i], salt) == hash:
                keyboardDict = keyboardDictOrig
                encodedPwd = encodedPwd[0:i]
                return keyboardEncToAscii(encodedPwd[0:i])



print("Master Password Generator for Phoenix BIOS (five decimal digits version)")
print("Copyright (C) 2009 dogbert <dogber1@gmail.com>")
print("")
print("After entering the wrong password for the third time, you will receive a")
print("decimal number from which the master password can be calculated,")
print("e.g. 12345")
print("")
print("Please enter the number: ")
code = raw_input().replace('[', '').replace(']', '')
hash = int(code)
print("")
print("Brute forcing passwords...")
print("Generic Phoenix BIOS:          " + bruteForce(hash, 0))
print("HP/Compaq Phoenix BIOS:        " + bruteForce(hash, salt=17232))
print("FSI Phoenix BIOS (generic):    " + bruteForce(hash, salt=65, minLen=3, maxLen=7,digitsOnly=True))
print("FSI Phoenix BIOS ('L' model):  " + bruteForce(hash+1, salt=ord('L'), minLen=3, maxLen=7,digitsOnly=True))
print("FSI Phoenix BIOS ('P' model):  " + bruteForce(hash+1, salt=ord('P'), minLen=3, maxLen=7,digitsOnly=True))
print("FSI Phoenix BIOS ('S' model):  " + bruteForce(hash+1, salt=ord('S'), minLen=3, maxLen=7,digitsOnly=True))
print("FSI Phoenix BIOS ('X' model):  " + bruteForce(hash+1, salt=ord('X'), minLen=3, maxLen=7,digitsOnly=True))
print("")
print("done.")
print("")
print("Please note that the password has been encoded for the standard US")
print("keyboard layout (QWERTY).")
if (os.name == 'nt'):
    print("Press a key to exit...")
    raw_input()

Tuesday, 23 September 2014

geda PCB font utilities and options including hebrew, greek, cyrillic and unicode CJK glyphs

I have converted some of the free Hershey fonts for use with the free and open source geda PCB design software, after starting with Hershey Sans 1 stroke to provide a more contoured default font.

In the process I wrote a small piece of C code pcbFontTool.c to simplify processing of pstoedit outputs and conversion into PCB Designer compatible font symbols.

Having streamlined the process somewhat, I figured why not do some of
the other fonts, so, PCB users now have the choice of:

Hershey Sans 1 stroke
Hershey German Gothic
Hershey Italian Gothic
Hershey English Gothic
Hershey Cyrillic - with keyboard mapping to be determined by user
Hershey Greek - with keyboard mapping to be determined by user

and the usual default_font

I have uploaded the fonts, the pcbFontTool utility, and instructions for those wishing to convert other fonts, to an ftp server

The license for pcbFontTool.c is GPL v2  or at the user's option GPL v3. The code could be improved but it does the job quite effectively.

These fonts are a simple drop in replacement for default_font, which at this stage lives in /geda/pcb/src
Until we have a mechanism for selecting fonts in PCB, this is what users will have to do.

I strongly recommend that anyone using geda PCB install git, clone the git repository, and build a local version of PCB so that they are running the latest version of PCB.

The pcbFontTool.c utility is standalone, only uses stdio, and has an attached README.txt

users.on.net/~esh/geda/pcb/src/fonts/README.txt

The gothic and sans fonts use the normal ASCII/US keyboard mapping.

The cyrillic font uses a keyboard mapping based on the Hershey fonts as distributed.
There are four or more types of cyrillic keyboard mapping in use, so a glyph table has been included to assist anyone trying to use it.

Likewise for the Greek font, a glyph table is included and PCB users will need to determine how they want to map the symbol to their keyboard. Users will have to add their own accents to the Greek vowels. There is a glyph table to show the Greek mapping as well.
 
The issue of CJK (Chinese, Japanese and Korean) glyphs was looked at next.

The GNU unifont bdf is available from unifoundry, which has released the GNU unifont as a free and open source font for use in FOSS.

http://unifoundry.com/pub/unifont-7.0.03/font-builds/unifont-7.0.03.bdf.gz

hosted by:

http://unifoundry.com/unifont.html

The reason that the GNU unifont has been made available in bdf format is because the bdf format is commonly used by X windows systems

http://en.wikipedia.org/wiki/Glyph_Bitmap_Distribution_Format

The bdf font file is a series of consecutive symbol definitions of the form:

STARTCHAR U+004E
ENCODING 78
SWIDTH 500 0
DWIDTH 8 0
BBX 8 16 0 -2
BITMAP
00
00
00
00
42
62
62
52
52
4A
4A
46
46
42
00
00
ENDCHAR

Chinese, Japanese and Korean (CJK) glyph rendering is not trivial and the approach taken by others has been to use bitmapped glyphs for resolutions below 16x16 pixels, and vector font formats for larger sizes.

The GNU unifont project has an essentially complete set of CJK glyphs in 11x11, 12x12, 13x13, 14x14, 15x15 and 16x16 pixel bitmap formats as produced and released as a free and open CJK font by Firefly around 2005.

The GNU unifont project includes around 20,000 CJK symbols that can now be used in gEDA PCB if required.

The batch converted set of CJK symbols are based on the 16x16 bitmapped Firefly CJK font set and have had contiguous pixels vertically, horizontally and diagonally converted into SymbolLine[] strokes, and orphan pixels if any, are rendered as a dot. A default stroke width of 800 has been used.

The smaller 11x11, 12x12 13x13, 14x14, 15x15 bitmapped CJK glyphs also could be easily converted if necessary, but they may lack the fidelity of the 16x16 glyphs which can be scaled anyway within PCB, so I have not bothered to convert the lower resolution bitmaps at this stage.

I have finished off the pcbGlyphTool code I used to batch convert Chinese/Japanese/Korean (CJK) Firefly bitmap glyphs to pcb symbols

This more polished version of the code fixes a boundary condition affecting 2 or 3 glyphs in the Firefly CJK 16x16 font set with pixels along edges.

The pcbGlyphTool utility:

1) acquires a valid BDF symbol definition via stdin
2) extracts the glyph label, glyph height, glyph width, display width, and the bitmap nibbles
2.1) optionally exports an xbm bitmap
2.2) optionally exports a "Dot matrix" PCB symbol rendition of the glyph using SymbolLine strokes to depict dots
3) stores the nibbles for each line in the glyph as a single integer
4) creates arrays in which each pixel is depicted as an integer
5) steps through the single integer representation of the rows and scores each pixel, putting the score into the integer per pixel row array
6) exports consecutive rows of pixels as SymbolLine strokes
7) steps through each column of scored pixels and does further scoring of each pixel, putting the score into the integer per pixel column array
8) exports consecutive columns of pixels as a SymbolLine strokes
8.1) optionally exports a symbol without diagonal row detection and conversion to strokes
9) creates left and right skewed arrays of the final pixel scores after column and row export
10) steps through the single integer representation of the right skewed array columns and detects diagonals, exports SymbolLine strokes
11) steps through the single integer representation of the left skewed array columns and detects diagonals, exports SymbolLine strokes
12) identifies any left over/orphan pixels and exports a SymbolLine stroke to depict a "dot"
13) exports a complete symbol with vertical, horizontal, and diagonal strokes
14) looks for another BDF symbol via stdin
This produces output containing a series of geda PCB symbol definitions for each glyph, and is what is in the gz file linked to above.

Until we have a mechanism for seamlessly adding unicode symbols or rendering ttf fonts, users needing glyphs can search the gzipped archive and cut and paste their needed symbol, and relabel it to assign it to an unused ascii character.

The utility compiles easily with gcc and one need only feed it with a bdf symbol definition or a stream of bdf symbol definitions to produce PCB symbols for insertion into a PCB layout, with suitable re-labeling to comply with the current limitations imposed by the implementation of "Symbol" which only copes with ~ 127 symbols.

i.e.

cat mydesiredglyph.bdf | ./pcbGlyphTool > NewSymbolForPCB.pcb

./pcbGlyphTool -h

will list other options, including -d "dot matrix rendition", -o "omitting" conversion of diagonals into strokes, -v "verbose" mode, and -x generate an xbm mode.

It should convert any bdf formatted bitmap up to 16x16 pixels into SymbolLine strokes replacing contiguous pixels in horizontal, vertical or diagonal directions, and it will render remaining "orphan pixels" as a dot.

Users requiring a few glyphs can now include them and map them to spare ASCII symbols until there is a way to more easily include unicode symbols, i.e.

Symbol['6' 1200]
#gEDA PCB compatible symbol with drawn elements depicting uni9ED6
#Symbol['uni9ED6' 1200]
(  etc...
)
I expected to walk away from the computer for at least a few minutes to convert the 20,000+ CJK bdf archive but it was done in seconds, meaning that on the fly importing of single CJK glyph bdf definitions from the freely downloadable GNU unifont bdf could in theory be done from within PCB if a suitable menu option were available.

The entire set of batch converted 16x16 bitmapped CJK glyphs as an uncompressed text file weighs in at around 20MB, but compressed is only around 1.5MB

I hope this is useful to anyone desperately in need of some functional CJK glyphs before gEDA PCB supports either an integrated conversion process like this or TTF support.

I cannot vouch for the rendering of all of the glyphs, as I do not read Chinese, Japanese or Korean, I haven't had the chance to review them all, and the heuristics may have joined the occasional diagonally adjacent pixels which should remain unjoined.

The symbol archive is released under GPL2 or at your option, a later version, can be freely distributed, and the usual font exception applies.

I would make the observation that this approach to glyph rendering makes for quite a compact symbol definition. My initial efforts involving conversion of curved paths over the glyphs produced symbol definitions 3-4 times the overall size, with implications for final PCB file size. It would be interesting to see how much bigger or smaller gerbers might end up being if text is rendered as polygons derived from TTF fonts.

As a bonus, the same bdf glyph conversion tool pcbGlyphTool.c was able to generate a Hebrew font from the GNU unifont 16x16 pixel bdf file, and this is located at

http://users.on.net/~esh/geda/pcb/src/fonts/UnifontDerivedHebrew_Font

and a glyph table is included so that users can decide how to map them to their keyboard

http://users.on.net/~esh/geda/pcb/src/fonts/gEDA-PCB-Unifont-Hebrew.pcb

The hebrew font could no doubt be improved aesthetically, but it is a start at least.

Anyway, someone may find the utilities or their subroutines useful.

I make no claims as to my code's elegance or fitness for purpose; the production of the code was a learning exercise and if I were to do it again I would of course do it differently...