I have a question related to the tweaking of TCP timeouts and retries. Basically I have an application that connects to a remote server to perform various operations. If the server dies however, it will perform operations on a local database then sync up with the server later. In linux, if the remote server is offline, the application tries to reach the server for approximately 2 minutes before it gives up and uses the local database. This of course makes the application seem like it has hung.

I’ve done some research into tweaking tcp settings. I’ve found various settings that I can tweak in the proc systems using sysctl. I’m not sure which ones will work for this task. I’m currently playing with:

tcp_orphan_retries
tcp_retries1
tcp_retries2

Does anyone know if I’m on the right track? Also I know these flags alter the settings based on the retransmission timeout (RTO). Is there anyway of tweaking the RTO? Or will tweaking one of the above flags suffice?

My reply:

I think you are doing it the hard way…

The most common way to get around long connect times is to set the socket to non-blocking, call connect(), then call select() on the socket with a reasonable time-out, checking for errors and writability. If the select times out or shows a socket error then you know the connect() has not completed and you can close the socket. If select() returns that the socket is writable then the connect() was successful.

I first began programming back in 1978 using BASIC.  There were a few books available with sample programs, and I learned BASIC and assembler.  But I was also lucky  because a fellow calculator toting math geek (my TI-30 was kept in a  denim-look case which attached to my belt; very hip for 1978) had some sort of access to source code from a university.  The BIG software project that distracted us for months was a Star Trek game, which we modified weekly.  My friend told me about all the free software available at colleges and universities; some of it in cool languages like Pascal and C.

After taking some college courses, which included Pascal and Image (a database for HP minicomputers), I hardly programed at all for 10 years until I got an AT clone running DOS and Windows 3.1.  I briefly programmed with Turbo Pascal, but then decided I wanted to jump into C headfirst.  But how?  I did not have the time or funds to take formal courses, so I turned to free/open source code which was available on my local BBS (Bulletin Board System).  It was great!  With the free DJGPP C compiler for DOS I was able to learn by doing.  I also purchased a special introductory version of Visual C++ that just built Win16 programs, but I never used it much.

I was able to quickly learn about inlined assembly, VESA frame-buffer graphics, OpenGL, IPX and TCP/IP networking, and finally speech compression using free and open source code software.   So you might say I am making software available free as payback.  I like to think I am providing stepping stones for others to go further than I have, and just maybe someday they will provide stepping stones I can use to go even further myself.

Now to the first question: Why do I use the GPL?  The main reason is that it is extensively used, and it has been tested in court.  While I hope to never need to enforce a free software license in court, the fact that the GPL has been found to be enforceable adds a little peace of mind.

Another, somewhat selfish, reason is that the GPL is considered to be non-friendly to commercial software.  While this is based more on ignorance than on truth, the fact is that is the way it is seen.  But this allows me to periodically dual-license some of my code which pays to keep this site up and allows me to purchase software and hardware to support current and future projects.

In conclusion, while I promote and contribute to free and open source code projects, I still believe there is a place for commercial software.  And right now the GPL is the best license I can use to pay back to the free/open source code community, yet still make a little money.

Phil Frisbie, Jr.

Phil Frisbie, Jr.

My C program needs to send UDP packets in burst. I have the following code:

send_data()
{
err = sendto(sock, &msg1, len1, &add, lenadd);
err = sendto(sock, &msg2, len2, &add, lenadd);
err = sendto(sock, &msg3, len3, &add, lenadd);
err = sendto(sock, &msg4, len4, &add, lenadd);
err = sendto(sock, &msg5, len5, &add, lenadd);
}

I do not get any error.

When I look on the network with a sniffer, sometimes, I [see] packets. but the missing packets are never put on the network.

How can I lose packets if I have a private network with only two nodes?

I use setsockopt(sock,SOL_SOCKET, SO_SNDBUF, &maxsize, sizeof(int)); function to be sure I have enough buffer memory, but that not seams to help.

I have the same result with Windows or Linux.

Does someone have an idea?

My obnoxious reply [I guess I used to be grumpier than I am now!]:

Repeat after me: UDP is not reliable. UDP is not reliable……

REALLY! The TCP/IP stack can drop UDP datagrams at ANY time, even if there is buffer space, even on a loopback address! The sooner you understand and acknowledge this the less frustration you will experience.

JavaScript or Flash in themselves are not really the problem; it is how they are used.  I am most familiar with JavaScript, so I will use it for my examples.  I suppose you can also abuse Silverlight, but I am not familiar with people using it for dynamic menus.  Basically, your menus should either gracefully degrade when scripting is not available or you should provide an alternate menu.

There are some very cool JavaScript menus.  Most work by dynamically hiding/displaying links based on what your mouse pointer is hovering over.  If the menu is made up of an unordered list, or nested lists, the menu will likely expand so that all choices are visible when JavaScript is disabled.  The menu may look ugly, but it will work!

Other menus are actually based on CSS – the CSS controls the hiding/displaying – and JavaScript is just an enhancement.  The State of California template does it this way.

Before you choose a JavaScript menu try disabling JavaScript in your browser to see what happens.  If it does not degrade to something usable you either need to keep looking or add an optional menu in a noscript tag.  Personally, I would avoid the noscript tag option because it means you need to maintain another menu.

The last thing you need to do is to insure the menu is accessible without a mouse (both with and without scripting).  Try using just your tab and enter keys to navigate your site; you just might be surprised how hard or tedious it is!  While it is good practice to have your most used links near the top, or beginning, of your menu, it is critical for keyboard-only users.

I will talk more about menus and accessibility in a future post.

I would like to use your library HawkNL (or HawkVoiceDI) in a commercial application.  Does the LGPL allow me to do this? Or can I license your library for a commercial application?

My reply:

First, please read he full terms of the LGPL Version 2.  Basically, if you use the library as a Windows DLL or *nix loadable module or shared object, then you only need to provide source for the library and any modifications you added.  The end user MUST be able to modify the library source and recompile the library if they choose.  A good example of this was Sony’s use of HawkVoiceDI in the game Planetside.  Sony used HawkVoiceDI as a DLL and provided a link to the source code in the online manual.

And yes, these libraries, and some of my other code, is available under a commercial license.  Hawk Software is a part-time job for me, so licensing helps keep this website up!

I know some are waiting for the rest of the floating point to fixed point series, and I am sorry that it is taking a lot more time than I anticipated.  I had outlined the entire series and completed the first two parts before I began posting, and I thought that a week between parts would be sufficient to get through it, but I was wrong.  When dealing with code originally converted seven years ago my memory takes extra time to retrieve all the details.  And some intermediate code versions are missing, so I am recreating some intermediate code to properly show the steps.

Also, some steps are so detailed that I will just provide an overview in this series, and will post more detailed explanations in future posts.

For these reasons I always validate my pages as they are being developed, and occasionally I perform spot checks later.  So I was VERY surprised when I validated my last blog entry and discovered it was no longer valid XHTML Strict due to an attribute I had never heard of:  ‘aria-required’.The W3C Validator was not much help, it simply said ‘there is no attribute area-required’.  So on to Google…

ARIA stands for Accessible Rich Internet Applications, and it is currently a W3C draft. It was created by the Web Accessibility Initiative (WAI).  The ARIA specifications are designed to allow an author to properly convey user interface behaviors and structural information in document-level markup, to assistive technologies (browsers).

I then found this is a common issue with current WordPress themes.  The issue is within the templates for comments, the comments.php and comments-popup.php files.  If the author name and email address are required then the template adds the area-required attribute to these two text  input fields.  This may at first seem redundant; the template will also add the text (required) to those fields.  However, by including the attribute the assistive browser can insure the user completes those fields before they are allowed to submit the form, helping to eliminate the  confusion of an error box or error page.

The  bottom line is this is a great addition for web accessibility, and I commend the creators of the current WordPress theme templates, but the ARIA specifications are still draft.  As a purist when it comes to standards, it puts me in a though spot: do I leave it and live with pages that do not pass validation; do I use a JavaScript hack to add the attribute to the input tags so that the Validator is happy (but the final XHTML presented to the browser will still not be valid); or do I remove it?

Well, for now I have decided to leave it in the hope that the ARIA specification will be adopted soon.  I think if it helps just one of my users then it will be worth it.  But that does not mean I will stop looking for other alternatives to pass validation!

I would have linked it, but I did not want to take the time to get permission first! You think I am kidding?  Here is section 11 in its entirety (emphasis is mine):

11. Linked Sites
Our Links to Other Sites: Our Site may contain links to Web sites of third parties. We provide these links as a convenience, but do not endorse the linked site or anything on it. While their information, products, services and information may be helpful to you, they are independent entities and we do not control or endorse them. You agree that any visits to linked sites are at your own risk and governed by their privacy policies (if any).

Your Links to Our Site: You are not permitted to link or shortcut to our Site from your Web site, blog or similar application, without obtaining prior written permission from us.

I have read many stories over the years about how Bill did not really understand the Internet back in the mid-to-late 90s, but if he approved this then it shows he still doesn’t understand.  Linking is at the core of the web; it is expected, even courteous, to link to sites that are mentioned.  I am not happy when I read an article that mentions another site/company/product but does not include a link so I can get the direct information without added opinion.  I cannot think of a good reason for requiring permission other than control.

Then there is the rather funny section 17:

17. System Requirements
Depending upon what you would like to do with the Foundation (e.g., if you apply for a job or grant or if you want to download certain items), you may need to make adjustments to your computer hardware or software so that you will be able to view or receive documents as intended by us. To deal with us electronically, you agree to obtain or use a system at least meeting these requirements:

Basic Requirements

1. a computer, monitor, cable or modem allowing Internet access or direct-dial accessibility, and a printer capable of printing documents (and website screens) or a hard drive capable of storing data;
2. one of these computer operating systems:
   a. Windows NT
   b. Windows 2000
   c. Windows XP
   d. Mac OSX 10.4.4
3. any of these Internet browsers:
   a. IE 5.5 (for Windows NT, Windows 2000)
   b. IE 6.0, IE 7.0 (for Windows XP, Windows NT, Windows 2000)
   c. NS 7.1 (for Windows XP, Windows NT, Windows 2000)
   d. AOL 7.0, AOL 8.0 (for Windows XP, Windows NT, Windows 2000)
   e. Firefox 1.5, Firefox 2.0 (for Windows XP, Windows NT, Windows 2000, Mac OSX 10.4.4)
   f. Safari 2.0 (for Mac OSX 10.4.4);
4. this software: Adobe Acrobat Reader 7.0 or Adobe Acrobat 5.0 or higher;
5. a printer of the ability to make, download or store an electronic copy of any contracts or legal disclosures that we may provide (such as in connection with a job application).

We reserve the right to change system requirements at any time without prior notice. If we were required to obtain your consent to dealing with us electronically and we change our system requirements, we will give notice and a right to withdraw your previous consent, if and as required by applicable law.

Is that VERY specific, or not?  When you are too specific you must regularly update your information or else it looks out of date and amateurish.  OK, it DOES say you need to “use a system at least meeting these requirements.”  So my Windows 7 system meets the qualifications, but does my Ubuntu 10.4 system with Firefox 3.6?  I think so, but does Bill?

There is more if you decide to read it for yourself, but at over 5,400 words it may put you to sleep

Anyway…I am still working on a SIMPLE terms of service, which includes only what is necessary and cannot be used as a sleep aid!

I begin the conversion process by adding these macros to the source files to emulate the fixed point math functions which will be used later:

#define fixed32         float
#define ftofix32(x)     ((float)(x))  /* float value to fixed */
#define itofix32(x)     ((float)(x))  /* integer value to fixed */
#define fixtoi32(x)     ((int)((x) < 0.0f ? (x) - 0.5f : (x) + 0.5f))  /* fixed value to integer with rounding */
#define fixtof32(x)     (x)  /* fixed value to float */
#define fixabs32(x)     fabs(x)  /* absolute value */
#define fixmul32(x, y)  ((float)(x) * (float)(y))  /* multiply x and y */
#define fixdiv32(x, y)  ((float)(x) / (float)(y))  /* divide x by y */
#define fixadd32(x, y)  ((float)(x) + (float)(y))  /* use only where you suspect large values will be accumulated */
#define fixsub32(x, y)  ((float)(x) - (float)(y))  /* same caution as add */

First go through your code and replace every instance of ‘float’ with ‘fixed32′:

void hp100(fixed32 *speech, long end, struct lpc10_encoder_state *st)
{
    fixed32 z11;
    fixed32 z21;
    fixed32 z12;
    fixed32 z22;

Then go through and modify each use of a static float or integer (integers only if they are used in floating point math equations) to use the macros:

static fixed32 enscl[8] = { ftofix32(.0204f),ftofix32(.0167f),ftofix32(.0145f),ftofix32(.0147f),
ftofix32(.0143f),ftofix32(.0135f),ftofix32(.0125f),ftofix32(.0112f) };

static fixed32 vdcl[10] = { itofix32(600),itofix32(450),itofix32(300),itofix32(200),itofix32(0), itofix32(0),
itofix32(0),itofix32(0),itofix32(0),itofix32(0) };

Convert casts to use the macros:

minamd = fixtoi32(amdf[*minptr]);

Then convert each fabs, multiply, and divide:

ivrc[1] = fixdiv32((r[2] - fixmul32(ivrc[0], r[1])), (r[0] - fixmul32(ivrc[0], r[1])));

Now look carefully through your code and look for other floating point functions used. Add macros as necessary and convert the code to use them:

#define fixsqrt32(x)    (float)sqrt(x)
#define fixexp32(x)     (float)exp(x)
#define fixlog32(x)     (float)log(x)
 

Regularly make sure your floating point code still works while converting it to use the macros. Use the test framework or reference data you created in part 2. At this point your code should still be bit-exact.

NOTE: It is possible that using these macros will cause some floating calculations to no longer be bit-exact. It is common for x86 compilers to use 80-bit registers for intermediate results for increased speed, and then round/truncate back to 32 bits just before storing the values. These macros may reduce the use of the 80-bit registers by causing rounding/truncating more often. The differences will be insignificant for most calculations, but they can be noticeable in a loop which performs a multiply and accumulate for thousands of values. It is not really that these macros are reducing the precision of the calculations; it is that they are only allowing it to be at 32-bits and not higher.

Now it is time to determine the fixed point precision and perform scaling as needed. At this point my instructions will become a little fuzzy due to differences in compilers and platforms. How easy your own conversion is from here on will be determined by how well you really understand the code you are converting and how well you command the Force

Remember from Part 1 that our fixed point values are using a 32 bit integer partitioned into bits for the whole number (and sign) and the rest of the bits for the fraction. I usually begin with 16.16 fixed point (16 bits for each), but it depends on how large your large values are compared with the precision needed for the smallest values. It is a balancing act. Sometimes you are lucky and your code is only dealing with large values or very small values, but often different parts of your code are using both. To see how large your values are first add these two additional defines:

#define precision       16 /* bits for the fractional part */

#define floatlimit      (float)(((1 << (32 - precision)) / 2) - 1 )

Then replace your macros with these functions:

fixed32 fixmul32(fixed32 x, fixed32 y)
{
    fixed32 temp = x * y;

    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixdiv32(fixed32 x, fixed32 y)
{
    fixed32 temp = x / y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixadd32(fixed32 x, fixed32 y)
{
    fixed32 temp = x + y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}
fixed32 fixsub32(fixed32 x, fixed32 y)
{
    fixed32 temp = x - y;
    if(temp > floatlimit)
    {
        return floatlimit;
    }
    else if(temp < -floatlimit)
    {
        return -floatlimit;
    }
    return temp;
}

The code above works for me with Visual Studio; I simply set up breakpoints on all the appropriate return lines to detect overflow. When the code hits a break I look back through the stack trace to see where the overflow occurred. You can also change these functions to defines and use asserts or print out the locations of the overflows.

In the new defines we just added I began with 16-bits of precision for the fractional part of the fixed point values. If you are getting overflows reduce the fractional precision, one bit at a time, and retest until you either eliminate the overflows or fail your test. If you can back off the fractional precision to eliminate the overflows, and your code still passes your tests, then you are one lucky developer! But more likely when you reduce the fractional precision enough to eliminate the overflows your code will be broken.

Part 4 will cover scaling and modifying algorithms in preparation for the actual fixed point conversion