Diff is available on Unix-like systems and on Windows as WinDiff (supplied with some versions of Microsoft Visual Studio) or WinMerge.  Diff is short for difference; open up two text files and diff will immediately show you what lines are different between them.  Programmers use diff all the time.  For example, I will compare the current version of source files to the last version to make sure all the changes were documented.  Or another programer might submit an updated file to me, and I can diff my copy to the submitted copy to see exactly what was changed before I accept the changes.

A search-and-replace tool can find text in multiple files, and replace it with something else.  It is like the find-and-replace you can perform with most text editors, except you can replace that text in hundreds or thousands of text files at once.  For example, as a programmer I might have a function named cleanupString(), but as the code matures that function is actually acts more like validateString().  I can use a search-and-replace tool to change that function name quickly in all my source files without opening each one up.

Now, how can these tools this help web developers?  Let’s start off with my recent use of diff.  I have a friend who is creating dozens of web pages for her work-group using a Dreamweaver template supplied by her organization. The template uses sever side includes (SSI) to add consistent headers and footers on all pages.  She had been doing OK until quite suddenly all her pages became ‘broken’.  She consulted with her IT department’s webmaster and was told she had altered the template and would need to start over and recreate all her pages!

She was not looking forward to the days of work ahead, but I told her to recreate just one page, and then send me a copy of that page (and all the includes and style sheets) and the broken page.  I used WinDiff to quickly see that Dreamweaver had modified one of the include files so that it was a stand-alone web page, which ‘broke’ all her pages.  I showed her how to fix the page by removing the extra HTML code from the include file, and her pages were fixed!

For a search-and-replace example: I took over a website years ago which had over 150 pages that were completely static; no use of SSI at all.  The pages had been created over a period of about five years, and some inconsistencies had crept in.  Menus, headers, and footers varied slightly from page to page, and to make things worse, the organization was going to adopt a new template within a year.

I used BK ReplaceEm (now called Replace Text) to clean up the inconsistencies and make use of SSI like the new template would.  The result was easier to maintain pages, and then when it was time to convert to the new template I did that with the help of BK ReplaceEm in just a couple of days.

These two tools have saved me hundreds of hours over the last 5 years, and they can save you time too.

So, how does this effect HawkThreads?  I had been planning to deprecate the HawkThreads API because I felt that yet-another-thread-library was not in the best long-term interest of my projects.  I had planned to just use Posix threads, and force Windows developers to use Pthreads-w32 (a very good implementation, but it requires an extra DLL), but I have other thoughts now.

As I looked at the C11 thread API, it looked very similar to the simple HawkThreads API.  C11 has threads, mutexes, conditions, and thread specific storage functions.  C11 has more functions, like timed conditions and locks, but it should not add that much more code.

C11 API:

void call_once(once_flag *flag, void (*func)(void));

int cnd_broadcast(cnd_t *cond);

void cnd_destroy(cnd_t *cond);

int cnd_init(cnd_t *cond);

int cnd_signal(cnd_t *cond);

int cnd_timedwait(cnd_t *cond, mtx_t *mtx, const xtime *xt);

int cnd_wait(cnd_t *cond, mtx_t *mtx);

void mtx_destroy(mtx_t *mtx);

int mtx_init(mtx_t *mtx, int type);

int mtx_lock(mtx_t *mtx);

int mtx_timedlock(mtx_t *mtx, const xtime *xt);

int mtx_trylock(mtx_t *mtx);

int mtx_unlock(mtx_t *mtx);

int thrd_create(thrd_t *thr, thrd_start_t func, void *arg);

thrd_t thrd_current(void);

int thrd_detach(thrd_t thr);

int thrd_equal(thrd_t thr0, thrd_t thr1);

void thrd_exit(int res);

int thrd_join(thrd_t thr, int *res);

void thrd_sleep(const xtime *xt);

void thrd_yield(void);

int tss_create(tss_t *key, tss_dtor_t dtor);

void tss_delete(tss_t key);

void *tss_get(tss_t key);

int tss_set(tss_t key, void *val);

int xtime_get(xtime *xt, int base);

So, on top of the other code I am working on, I will be creating a major 2.0 update to HawkThreads to provide a (mostly) C11 interface so that it can be used on platforms that do not yet have a C11 compliant compiler.  I say mostly because I am not fully sold on the way conditions are handled in C11 (or in Posix for that matter).  Perhaps I just need to understand the choice of including a mutex when signalling and waiting on a condition.  I will study it more and get back to you. . .

It is interesting that I had already discovered a way to avoid the 64-bit issues in my preliminary work on HawkFloat, so that part is easy.  I think it is also finally time to switch over to the GPL 3.o, and all my new projects will use this almost 5-year old license (how time flies!).

Now to HawkNL 2.0:

NL 2.0 will have several changes, which will include removal of features which have crept in but are better to be in the utility library, HawkNLU, new layered source code so that all platform specific code will be in a single header file, IPv6 support, and a simpler API.

Some features which will be moved to NLU are reliable packets and socket statistics.  These are really too high a level for NL, and their removal will reduce the need for threading API use to just mutexes.

Some features to be removed altogether are the never completed serial, modem, and parallel drivers, and an option to remove the loopback driver since most people prefer to use the IP driver with localhost.  I plan to also remove the IPX code, and simply let people use the 1.X series if they really want IPX.

More details will emerge as coding continues. . .

A floating point number is composed of three parts: the sign bit, the exponent, and the mantissa (or significand). I will represent these as S, E, and M, respectively. The IEEE float is composed of a sign bit, 8 bits for exponent, and 23 bits for the mantissa (1S-8E-23M). The IEEE double float is 1S-11E- 52M, and the IEEE half float is 1S-5E-10M.

The Small Floating Point (SFP) format is 8E-1S-7M (note the order). It preserves the 8 bits of exponent from the IEEE float, and moves the sign bit next to the mantissa, thus creating a natural break along a byte boundary. Preserving the byte boundaries helps to reduce the number of masking and shifting operations in a software implementation.

I am proposing two floating formats: Hfloat, 7E-1S-8M; and Hdouble, 7E-1S-16M. Yes, that second format is just 24 bits (the first 8 bits are unused/ignored), but my reasoning will become apparent as you read on. In these new formats, the exponent is signed (not a biased unsigned number like in IEEE formats), and it is at the front so a simple shift will pack or unpack it while preserving the sign. Converting from an Hfloat to an Hdouble is a simple shift. The mantissa does NOT include an implicit leading 1, which allows us to do away with the special case of denormal numbers and the associated code complexity at the expense of 0.3 bits of precision (see chart below).

Float      Mantissa   Digits of

Format       Bits     Precision

IEEE half    10+1       3.3

IEEE float   23+1       7.2

SFP          8+1        2.7

Hfloat       8          2.4

Hdouble      16         4.8

The Hdouble is just that; double the digits of precision compared to an Hfloat. Also, by limiting it to 16 mantissa bits, only a 32-bit integer is needed to perform a multiply, resulting in faster performance and less code. Since the exponents are the same, there is more code reuse, which will result in fewer lines of code and fewer potential bugs.

I said there were two new float types, the Hfloat and the Hdouble, but there is actually a third type that can help in common digital signal processing (DSP): the Haccumulator, which is used as an accumulator for a multiply-accumulate function. It is a structure which contains a byte and a 32-bit integer to hold 7E-1S-32M, and it will be converted to an Hfloat or Hdouble before further use.

The last thing to mention in this post is that there is no support for the IEEE NaN, positive and negative infinity, or exceptions. All functions saturate (to the maximum or minimum value), which helps to eliminate divide by zero (which will be equal to the max value if it does occur). Now that I have the specs flushed our fairly well, it is time to begin coding. . .

I am not going into a description of floating and fixed point math right now, so if you are not familiar with them you can read this floating point article and/or this fixed point article.  For this post, I will begin by highlighting a few of the reasons software floating point math, specifically IEEE 754 conforming floating point, is so slow in software.

The IEEE 754 standard was originally designed to support 32 and 64 bit floating point numbers, and the spec is very tight to insure that calculations on any supported platform will result in the same values.  It was also created with scientists in mind, who often deal with very small and very large numbers.  The spec provides for special values including negative and positive infinity, NaN (Not a Number), and denormal numbers.  There are are also several rounding modes which must be supported.

Implementing all the above takes a lot of code, and requires many special cases which slows down calculations.  Also, for many signal processing tasks involving image and sound processing, the range of even a 32-bit float is more than needed.  This was partially addressed in the latest IEEE 754-2008 revision which support 16-bit floats.  However, these new 16-bit floats still have all the overhead of the original spec, and they do not have a very large range; they can only represent values up to +-65504, or just less than an unsigned short integer.

In my research I have found a library which is close: Small Floating Point (SFP).   At first glance it looked like it could be a great start.  The library uses a modified 16-bit format, which is designed on byte boundaries to reduce masking and shifting, and it has increased range at the expense of some precision.  However, the code uses something called “Primitive C”, which is hard to read, but intended to be easier to port to assembly.  The code also uses shift based multiply and divide because it is targeted for 8-bit CPUs.

So, next time, I will outline my goals for a new 16-bit floating point library that targets modern 32-bit ARM CPUs that have hardware integer multiply and divide.  I will also talk about the precision/range tradeoffs I went through in choosing my 16-bit float format, and how it affects final code.

So, now that my forum status is officially Addict, I will try to back off a little and spend more time on programing.  I think I have some interesting things in store for you.

First, there are some speech codecs which are now likely unencumbered by Intellectual Property (IP) rights, in other words, patents. Many speech related patents have been expiring over the last few years.  Others, like GSM Half-Rate, have IP that is either tied to the GSM cellular network protocols, or are so vague they are likely unenforceable; more about that another time.

Second, I have been rethinking fixed point math.  It is just TOO hard to convert anything other than small trivial programs from floating to fixed point so they run on ARM and other CPUs without hardware floating point (also called vector instructions).  There is a LOT of time involved converting code, testing, tracking down under and over flows, etc. Yes, I can do it, and have done it, but I am not sure I want to spend the time to converting another project. However, there ARE other ways to speed up floating point calculations by an order of magnitude; more on that another time.

Third, I still love 3D graphics.  I miss programming in OpenGL, but I want a useful challenge also. Despite the ever increasing availability of 3D hardware, even on such things as phones, I still feel there is a need for a small fast software OpenGL library. My inspiration is from  TinyGL, but mine might be more like OpenGL ES.  I also envision  some options like no copy of textures to reduce memory use, and to force all memory allocations to be approved and made by the application.  More on this also.

Fourth, yes, I have disabled comments.  The bots were becoming annoying, and you can easily email anyway.  So don’t be shy, send me an email, and tell me what you think about some of these potential projects.

I began HawkNL back in 2000 with the goal to produce a cross platform, and easy to use, alternative to Microsoft’s DirectPlay.  Developers liked it, but some asked about DirectPlaySound, which allowed a speech channel to be added to a DirectPlay connection. So, within months, I was searching for free and open source speech codecs.  The high level API, HawkVoice, was to be dependent on HawkNLU, which was to be built on top of HawkNL.  In order for people to directly use  the HawkVoice codecs I created HawkVoiceDI (Direct Interface), which has been used in dozens of projects.

Now, here we are, DirectPlay is deprecated, and most people are using their own frameworks for network communications, so HawkVoiceDI will now become HawkVoice, and it will simply be an easy to use speech compression library with a few helper functions to help add voice chat to applications.  The larger goal of integration with HawkNLU is now history.

Now the main focus of HawkVoice will be the codecs, with a little work on helper functions.  I am looking at improving the current codecs, and adding some more.  For example, years ago when I added GSM I was more worried about the total cycles per second for encoding/deciding, and if the sound quality was not noticeably degraded it was OK.  I will start off fresh with reference GSM code and improve it ONLY WHILE THE OUTPUT IS STILL BIT EXACT.  The same goes for LPC-10, and perhaps others.

I am also seeing that more people want to use wide-band codecs (16 kHz sample rate) rather than narrow band (8 kHz sample rate), and I have two codecs I am checking right now that look promising.  Of course, hand held devices are really taking off, so all new codecs will be fixed point to run on ARM CPUs.

I will be updated the HawkVoice web pages soon with more details.

No, it was Microsoft’s description of the new native Win8 apps, which are built on HTML 5 and JavaScript. HTML 5? JavaScript?? And I thought Microsoft was pushing HTML 5 compliance and JavaScript performance just for the web! Now we know the REAL story; it is to speed up native Win8 apps. Microsoft can finally realize it’s dream of applications being interpreted. No, it’s not NET like they originally pushed, but with JavaScript virtually ALL web developers will be able to create native Win8 apps.

This is a strong, but risky move; not the type that Microsoft has been know for in recent years. The potential positive aspects I see are the huge increase in the developer talent pool I already mentioned, increased app security, lower development costs, easier to target desktop-pad-phone-console, and easier app store integration.

However, the negatives for Microsoft (and a boon for others) could be easier to port apps to other platforms (Linux, Android, MacOS) and consumer backlash against the new UI. Of course, we have no idea yet how Microsoft will package Win8 apps, or how they might restrict them. And perhaps consumers have become so used to their portable devices that they will welcome the same interface on the desktop.

There is speculation Microsoft will simply use HTML Applications (HTAs), which have been supported since IE 5, but have remained fairly obscure. Since all but trivial apps will require various resources, images, sounds files, etc., they may already have an archive container chosen to more easily package everything up. Whatever the format, unless they have figured out some way to lock in developers to Win8, perhaps some mandatory copyrighted JavaScript classes, developers will easily create the same functionality on other platforms. And I wonder how long before other browsers decide to support HTAs?

I have been saying for years that JavaScript was under appreciated, and the great speed increases in today’s browsers has helped to keep it as the #1 language of the web (you thought that was Java?). I look forward to using JavaScript in desktop apps.

Yes, Win8 really surprised me, but I can’t wait to see if Microsoft is the one surprised in the end.

Personally, I would be happy with either WebM or a REALLY free version of H.264.   If the MPEG LA really wants H.264 to become the  standard then they should allow it to be used forever for free on the web, both for encoding and decoding.  That would end the war right now, and relegate WebM to a blip in web history.

What really bothers me, however, is all the misinformation pro-H.264 experts are distributing. For example, the article which sparked this post  is in SD Times, a magazine I read from cover to cover.  The February 2011 issue has an article on the current war, and I cannot help to comment on some remarks that IDC program director Al Hilwa said.

First, Hilwa begins by stating that Google’s motivation to support only WebM is all about license fees, which perhaps is true, but ignores Google’s past support of free software in general, and the freedom which comes from free and open source software.

Then he really shows his agenda by stating “They want unencumbered technology, which is an idealistic view that doesn’t square with our system of intellectual property.”  Software patents are another issue, and if it is truly not possible to create a codec free of patented technology then it is another sign that our software patent system is broken.  He then says WebM is only as good as the first patent lawsuit. Yes, hear comes the FUD (Fear, Uncertainty, Doubt)!

The final straw for me was when he asks the question “…will content providers such as Hulu and Netflix support an open codec, which might be easier to hack?”  I hope he was misquoted on that one, because rights management is totally separate from the codec.  Windows is closed source, and we all know how secure it is. . .