I have a question related to the tweaking of TCP timeouts and retries. Basically I have an application that connects to a remote server to perform various operations. If the server dies however, it will perform operations on a local database then sync up with the server later. In linux, if the remote server is offline, the application tries to reach the server for approximately 2 minutes before it gives up and uses the local database. This of course makes the application seem like it has hung.

I’ve done some research into tweaking tcp settings. I’ve found various settings that I can tweak in the proc systems using sysctl. I’m not sure which ones will work for this task. I’m currently playing with:

tcp_orphan_retries
tcp_retries1
tcp_retries2

Does anyone know if I’m on the right track? Also I know these flags alter the settings based on the retransmission timeout (RTO). Is there anyway of tweaking the RTO? Or will tweaking one of the above flags suffice?

My reply:

I think you are doing it the hard way…

The most common way to get around long connect times is to set the socket to non-blocking, call connect(), then call select() on the socket with a reasonable time-out, checking for errors and writability. If the select times out or shows a socket error then you know the connect() has not completed and you can close the socket. If select() returns that the socket is writable then the connect() was successful.

I first began programming back in 1978 using BASIC.  There were a few books available with sample programs, and I learned BASIC and assembler.  But I was also lucky  because a fellow calculator toting math geek (my TI-30 was kept in a  denim-look case which attached to my belt; very hip for 1978) had some sort of access to source code from a university.  The BIG software project that distracted us for months was a Star Trek game, which we modified weekly.  My friend told me about all the free software available at colleges and universities; some of it in cool languages like Pascal and C.

After taking some college courses, which included Pascal and Image (a database for HP minicomputers), I hardly programed at all for 10 years until I got an AT clone running DOS and Windows 3.1.  I briefly programmed with Turbo Pascal, but then decided I wanted to jump into C headfirst.  But how?  I did not have the time or funds to take formal courses, so I turned to free/open source code which was available on my local BBS (Bulletin Board System).  It was great!  With the free DJGPP C compiler for DOS I was able to learn by doing.  I also purchased a special introductory version of Visual C++ that just built Win16 programs, but I never used it much.

I was able to quickly learn about inlined assembly, VESA frame-buffer graphics, OpenGL, IPX and TCP/IP networking, and finally speech compression using free and open source code software.   So you might say I am making software available free as payback.  I like to think I am providing stepping stones for others to go further than I have, and just maybe someday they will provide stepping stones I can use to go even further myself.

Now to the first question: Why do I use the GPL?  The main reason is that it is extensively used, and it has been tested in court.  While I hope to never need to enforce a free software license in court, the fact that the GPL has been found to be enforceable adds a little peace of mind.

Another, somewhat selfish, reason is that the GPL is considered to be non-friendly to commercial software.  While this is based more on ignorance than on truth, the fact is that is the way it is seen.  But this allows me to periodically dual-license some of my code which pays to keep this site up and allows me to purchase software and hardware to support current and future projects.

In conclusion, while I promote and contribute to free and open source code projects, I still believe there is a place for commercial software.  And right now the GPL is the best license I can use to pay back to the free/open source code community, yet still make a little money.

Phil Frisbie, Jr.

My C program needs to send UDP packets in burst. I have the following code:

send_data()
{
err = sendto(sock, &msg1, len1, &add, lenadd);
err = sendto(sock, &msg2, len2, &add, lenadd);
err = sendto(sock, &msg3, len3, &add, lenadd);
err = sendto(sock, &msg4, len4, &add, lenadd);
err = sendto(sock, &msg5, len5, &add, lenadd);
}

I do not get any error.

When I look on the network with a sniffer, sometimes, I [see] packets. but the missing packets are never put on the network.

How can I lose packets if I have a private network with only two nodes?

I use setsockopt(sock,SOL_SOCKET, SO_SNDBUF, &maxsize, sizeof(int)); function to be sure I have enough buffer memory, but that not seams to help.

I have the same result with Windows or Linux.

Does someone have an idea?

My obnoxious reply [I guess I used to be grumpier than I am now!]:

Repeat after me: UDP is not reliable. UDP is not reliable……

REALLY! The TCP/IP stack can drop UDP datagrams at ANY time, even if there is buffer space, even on a loopback address! The sooner you understand and acknowledge this the less frustration you will experience.

I would like to use your library HawkNL (or HawkVoiceDI) in a commercial application.  Does the LGPL allow me to do this? Or can I license your library for a commercial application?

My reply:

First, please read he full terms of the LGPL Version 2.  Basically, if you use the library as a Windows DLL or *nix loadable module or shared object, then you only need to provide source for the library and any modifications you added.  The end user MUST be able to modify the library source and recompile the library if they choose.  A good example of this was Sony’s use of HawkVoiceDI in the game Planetside.  Sony used HawkVoiceDI as a DLL and provided a link to the source code in the online manual.

And yes, these libraries, and some of my other code, is available under a commercial license.  Hawk Software is a part-time job for me, so licensing helps keep this website up!

I know some are waiting for the rest of the floating point to fixed point series, and I am sorry that it is taking a lot more time than I anticipated.  I had outlined the entire series and completed the first two parts before I began posting, and I thought that a week between parts would be sufficient to get through it, but I was wrong.  When dealing with code originally converted seven years ago my memory takes extra time to retrieve all the details.  And some intermediate code versions are missing, so I am recreating some intermediate code to properly show the steps.

Also, some steps are so detailed that I will just provide an overview in this series, and will post more detailed explanations in future posts.

I begin the conversion process by adding these macros to the source files to emulate the fixed point math functions which will be used later:

#define fixed32         float
#define ftofix32(x)     ((float)(x))  /* float value to fixed */
#define itofix32(x)     ((float)(x))  /* integer value to fixed */
#define fixtoi32(x)     ((int)((x) < 0.0f ? (x) - 0.5f : (x) + 0.5f))  /* fixed value to integer with rounding */
#define fixtof32(x)     (x)  /* fixed value to float */
#define fixabs32(x)     fabs(x)  /* absolute value */
#define fixmul32(x, y)  ((float)(x) * (float)(y))  /* multiply x and y */
#define fixdiv32(x, y)  ((float)(x) / (float)(y))  /* divide x by y */
#define fixadd32(x, y)  ((float)(x) + (float)(y))  /* use only where you suspect large values will be accumulated */
#define fixsub32(x, y)  ((float)(x) - (float)(y))  /* same caution as add */

First go through your code and replace every instance of ‘float’ with ‘fixed32′:

void hp100(fixed32 *speech, long end, struct lpc10_encoder_state *st)
{
    fixed32 z11;
    fixed32 z21;
    fixed32 z12;
    fixed32 z22;

Then go through and modify each use of a static float or integer (integers only if they are used in floating point math equations) to use the macros:

static fixed32 enscl[8] = { ftofix32(.0204f),ftofix32(.0167f),ftofix32(.0145f),ftofix32(.0147f),
ftofix32(.0143f),ftofix32(.0135f),ftofix32(.0125f),ftofix32(.0112f) };

static fixed32 vdcl[10] = { itofix32(600),itofix32(450),itofix32(300),itofix32(200),itofix32(0), itofix32(0),
itofix32(0),itofix32(0),itofix32(0),itofix32(0) };

Convert casts to use the macros:

minamd = fixtoi32(amdf[*minptr]);

Then convert each fabs, multiply, and divide:

ivrc[1] = fixdiv32((r[2] - fixmul32(ivrc[0], r[1])), (r[0] - fixmul32(ivrc[0], r[1])));

Now look carefully through your code and look for other floating point functions used. Add macros as necessary and convert the code to use them:

#define fixsqrt32(x)    (float)sqrt(x)
#define fixexp32(x)     (float)exp(x)
#define fixlog32(x)     (float)log(x)
 

Regularly make sure your floating point code still works while converting it to use the macros. Use the test framework or reference data you created in part 2. At this point your code should still be bit-exact.

NOTE: It is possible that using these macros will cause some floating calculations to no longer be bit-exact. It is common for x86 compilers to use 80-bit registers for intermediate results for increased speed, and then round/truncate back to 32 bits just before storing the values. These macros may reduce the use of the 80-bit registers by causing rounding/truncating more often. The differences will be insignificant for most calculations, but they can be noticeable in a loop which performs a multiply and accumulate for thousands of values. It is not really that these macros are reducing the precision of the calculations; it is that they are only allowing it to be at 32-bits and not higher.

Now it is time to determine the fixed point precision and perform scaling as needed. At this point my instructions will become a little fuzzy due to differences in compilers and platforms. How easy your own conversion is from here on will be determined by how well you really understand the code you are converting and how well you command the Force

Remember from Part 1 that our fixed point values are using a 32 bit integer partitioned into bits for the whole number (and sign) and the rest of the bits for the fraction. I usually begin with 16.16 fixed point (16 bits for each), but it depends on how large your large values are compared with the precision needed for the smallest values. It is a balancing act. Sometimes you are lucky and your code is only dealing with large values or very small values, but often different parts of your code are using both. To see how large your values are first add these two additional defines:

#define precision       16 /* bits for the fractional part */

#define floatlimit      (float)(((1 << (32 - precision)) / 2) - 1 )

Then replace your macros with these functions:

fixed32 fixmul32(fixed32 x, fixed32 y)
{
    fixed32 temp = x * y;

    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixdiv32(fixed32 x, fixed32 y)
{
    fixed32 temp = x / y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixadd32(fixed32 x, fixed32 y)
{
    fixed32 temp = x + y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixsub32(fixed32 x, fixed32 y)
{
    fixed32 temp = x - y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}

The code above works for me with Visual Studio; I simply set up breakpoints on all the appropriate return lines to detect overflow. When the code hits a break I look back through the stack trace to see where the overflow occurred. You can also change these functions to defines and use asserts or print out the locations of the overflows.

In the new defines we just added I began with 16-bits of precision for the fractional part of the fixed point values. If you are getting overflows reduce the fractional precision, one bit at a time, and retest until you either eliminate the overflows or fail your test. If you can back off the fractional precision to eliminate the overflows, and your code still passes your tests, then you are one lucky developer! But more likely when you reduce the fractional precision enough to eliminate the overflows your code will be broken.

Part 4 will cover scaling and modifying algorithms in preparation for the actual fixed point conversion

• Associate Mutexes with Data to Prevent Races
• Numerical Recipes
• The TCP-Friendly Website
• Secure Programming.com
• rt_lpc : real-time LPC analysis + synthesis + visualization

When I get into winsock in globalization, I found out winsock.h has a lot of different data structure if _UNICODE turned on. But, there is no function support those data types. such as gethostbyname(…) only take const char FAR * name. How could I call gethostbyname when _UNICODE turned on in Visual C++?

My reply:

You do it like everyone else! You must provide code to convert from unicode to ascii and back into unicode whenever you deal with Winsock functions.

For example, my free open source code network library HawkNL compiles with or without unicode, so in my function to resolve a host name to an address I do this:

#ifdef _UNICODE
/* convert from wide char string to multibyte char string */
(void)wcstombs(temp, (const NLchar *)name, NL_MAX_STRING_LENGTH);
#else
strncpy(temp, name, NL_MAX_STRING_LENGTH);
#endif

So if unicode I convert the string, but if not I copy it before I process it further before I call gethostbyname().

Starting off with clean, fully debugged, fully functional, and stable floating point code is a must. Fixed point code can be harder to interpret, so you do not want to be modifying the converted code every time you correct or add a feature to the original code. It is even better to start off with code that has not been heavily hand-optimized since it will be easier to understand and convert. Of course, make sure your algorithms are efficient and you are not doing something stupid like scanning through arrays non-sequentially and getting a ton of cache misses

In preparing to convert the LPC-10 codec I reduced the number of .c source code files in the original code from 33 (it used one source file for each major function) down to just 2: one for encoding and one for decoding. I did this so that I could convert the encoder and decoder separately and make testing easier.

Now that you have clean code, you can create reference data and/or a test framework to verify your code’s correct operation. It is too easy to break your code while making the conversion, so you must be able to test it. In the LPC-10 codec I added code to dump the encoder output values to stdout where they could be redirected to a text file.

static void encode(long *voice, long pitch, float rms, float *rc, long *ipitch,
long *irms, long *irc)
{
…
printf("%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d,\t%d\n",
irc[0],irc[1],irc[2],irc[3],irc[4],irc[5],irc[6],irc[7],irc[8],irc[9],
*ipitch, *irms);
} /* encode_ */

To create my encoder reference values I encoded two known speech files using the unmodified floating point code and dumped the encoded values to files. It was then easy to regularly diff the reference files with the files generated by the fixed point code to see how accurate the conversion was. In the end, I stopped when the fixed point encoder achieved 95% accuracy compared to the floating point code, and the resulting decoded speech was indistinguishable from that generated by the floating point code.

For a lossy speech codec like LPC-10 I did not need to spend the time achieving 100% bit-exact results, but having the reference files helped assure me that my conversion was good, and helped to quickly notify me when I broke the code while trying to make it better.

Part 3 will cover the conversion process.

I have a question about the loss of UDP datagrams.  System setup like below:

 /-------------\    Crossover    /-------------\
 | System A    |  <------------> | System B    |
 | 192.168.0.1 |      Cable      | 192.168.0.2 |
 \-------------/      100MB/s    \-------------/

If I have a test program to send UPD datagrams across this connection, assuming I am not overloading the link, should I expect any datagram loss?

My reply:

Yes, with UDP you MUST expect lost packets and provide code to deal with them.

Not overloading the link has nothing to do with lost packets; timing is everything. For example, if the sender sends a burst of packets faster than the link layer can handle and the send buffer fills up then packets will be silently dropped. Or if the receiving thread gets delayed due to a momentary spike in CPU activity and cannot receive the packets fast enough and the receive buffer fills up packets will be discarded.