U-Streams

Integration time

30 minutes

Saves you

$6000 / 100h *

Description

U-Streams Library

The Problem

UNIX and its variants provide only stdout and stderr as standard write streams available to fprintf(). When wanting to direct messages elsewhere, it is necessary to make other arrangements for output; either via a command-line redirect, through internal management within a program itself, or some other (most-likely) non-trivial method (think pipes and sockets).

Modern systems, however, are very often required to send messages to multiple destinations, e.g., a disk-file, a database logger, alert and monitoring systems, auto-updating web-sites, UI displays, e-mail and twitter accounts, other programs, and the list goes on. When multiple destinations are necessary for the same message, this message needs to be duplicated for each destination, making coding annyoing, repetitive, and increasingly difficult to manage.

But what if there were a way to write a message just once in your program, while elsewhere it can be handled and forwarded to any number or kind of destinations without headache?

And wouldn't it be even nicer to be able to use newly named write streams that allow message type deliniation through more than just the two standard streams provided? For example, why not extend the standard stream concept to include stdInfo, stdWarn, and stdDbug?

The Solution

With this C source code library, a programmer can now:

Redirect stdin, forwarding all incoming messages to your registered callback handler.
Redirect stdout and stderr to multiple destinations, without having to rewrite any of your fprintf() statements.
Create new streams that behave identically to stdout and stderr, using the new stream name directly, just like the standard streams.
Define and register your own call-back routines, per stream, for program-specific handling of every message.

Features:

Unlimited number of new streams may be created
Unlimited number of callbacks may be registered per stream (and de-registered)
Registration and De-Registration of callbacks can be made at any point in your program; that is, they may be turned on and off throughout the course of a program's execution.
Caller can define and set the message delimiter, per stream
Works with background programs or GUI programs equally well
stdout and stderr streams originating from other libraries included in your program are also intercepted. For example, all MySQL error messages are sent through stderr and so could be intercepted by this library. This provides the opportunity for the programmer to act on these messages directly and immediately. For example, when receiving a crashed table message, your program could parse this message and automatically issue a repair table command, alleviating the user from having to do this themselves.
Works especially well when forking a program. Since all file descriptors are inherited by the child, all messages written by the child will be processed by the same callback routine, guaranteeing single concurrancy of message processing, i.e., no collisions regardless the number of children demanding attention.
It Is Extensible: In general, new streams can be created to be used as a channel to transmit anything to go anywhere and/or do anything.
- Example 1: Streams may also be created and used for other purposes. For example, counting an occurance of something, where this occurance happens in multiple locations of your code. In this situation, the callback maintains the system-wide counter, and can then carry out any necessary actions when the counter reaches a certain value. For example, a server program listening for client connections where connection management of all clients is a concern.
- Example 2: A new stream can be created that is responsible for executing one of many known tasks, essentially becoming a channel of "work". That is, anywhere in the program where a separate task must be performed, simply send a message through the stream and let the callback be reponsible for all logic and subsequent task execution.
- Example 3: And turning the concept around, it's possible to create streams not by message type, but by destination type. So, a stream may be designated to send information through the channel directly to a Database, a web-site, a twitter account, a UI display screen, a disk file, another program, a work queue, etc.
Creating a new stream is as simple as:
- Define the new stream name, in ustreams.c
- Add the extern declaration to ustreams.h file
- Add an equivalent USTREAMS_STREAM enum anywhere in the current list of known streams, in ustreams.h
- Add one line to the initialisation routine creating the stream itself, in ustreams.c
- Re-Compile the library
- Write the callbacks to handle the new stream's output
- New stream is now available for use everywhere in your program

Advantages:

Retrieve and process all stdin messages as a simple callback
Never have to deal with having to manage FILE* pointers, pipes, sockets, etc. directly in your app code ever again
As new output requirements come to exist, no need to re-write existing code, just write a new callback handler.
fprintf() now becomes a powerful tool of information transmission and routing; no longer just a statement to print a string
All logic associated with data of a stream is now centrally located and fully de-coupled from your program at-large
Using the stdDbug stream now simplifies necessary debugging statements. Write all your debug statements, and when you don't need the output, simply don't register a callback. And, since dynamic registration and de-registration of callbacks per stream is possible, it is trivial to isolate only those sections of the code requiring debug output for any given execution.
Ideal for large systems employing many libraries and/or having a large code base.
Because it's so easy to use/implement: If used during the Design Phase, new algorithms and solutions are now available, and trivial to implement.

Deliverables:

Source Code, Header Files, and makefile to create the ustreams.a library archive
New streams stdInfo, stdWarn, and stdDbug provided for
Complete Documentation describing all aspects of how to use the library, detailed instructions to create new streams, examples of extensibility,
A completed example implementation (skinny version below) demonstrating how to forward messages to:
- Multiple Disk Files
- A Database
- Email Addresses or Twitter Accounts

Target Users:

All UNIX/C developers interested in having precise control over program messages, their destinations, and any possible follow-on actions.
Any programmer wishing to extend the concept of a write stream beyond using it simply for string messages; the possibilities are limitless.

Technical Requirements:

Up-to-date C compiler
GTK+-2 installed on computer (though programming in GTK+-2 is not a requirement, everything is internalized). Gtk+-2 is standard on all *NIX platforms and should not be an issue.

Installation:

Library is provided as a single file source code, a header file, and externs header file for compilation.
Makefile is provided to create an archive named ustreams.a, to be used when linking the final program.

Sample Implementation - Described

The basic order of U-Streams functions:

ustreams_init() - Initialize the U-Streams library
ustreams_register() - register/de-register callbacks, per stream
ustreams_cmd() - issue a U-Streams command
ustreams_cmd(USTREAMS_SHUTDOWN) - U-Streams library shutdown

A sample coded implementation is below demonstrating:

System Initialisation
Callback Definition & Registration:
- Write to Disk Files
- Write to Database
- Send e-mail or tweet
Callback De-Registration of Disk File callback
System Shutdown

Sample Implementation - Code

#include 
#include 
#include 
#include 
#include "ustreams.h"

/*  
  Copyright Notice:
  Richard Ivor
  2012 - All rights reserved
*/

// our program name
static gchar *procName;

static gchar *dbPrefix[USTREAMS_TTL_STREAMS] = { "LOG", "ERR", "INFO", "WARN", "DBUG"};
#define USTREAMS_INSERT_SQL    "INSERT INTO `LOGS` (type, msg) VALUES ('%s', '%s')"
static void ustreamToDB(USTREAMS_STREAM stream, const char *msg)
{  // write message to DB, adding a descriptive prefix
  gboolean shutDown = USTREAMS_SHUTTING_DOWN(msg);
  gchar *dbMsg;

  switch (shutDown)
  {
    case FALSE:
    {
      gchar *dbMsg = g_strdup_printf(USTREAMS_INSERT_SQL, dbPrefix[stream], msg);

      // insert code to write the message to a Database
      // ...
      g_free(dbMsg);
    }
    break;
    case TRUE:
    {  // UStream is shutting down
      // Execute any shutdown/clean-up operations here, e.g., a DB disconnect?
    }
    break;
  }
  return;
}

static void ustreamToDisk(USTREAMS_STREAM stream, const char *msg)
{  // write messages to individual disk file based on stream, adding a prefix timestamp
  gboolean shutDown = USTREAMS_SHUTTING_DOWN(msg);
  gchar *diskMsg;

  switch (shutDown)
  {
    case FALSE:
    {  // construct and output the message
      diskMsg = g_strdup_printf("%s%s\n", timeStr, msg);
    }
    break;

    case TRUE:    
    {  // shutting down, make msg and close our disk file pointers
      // construct and output the message
      diskMsg = g_strdup_printf("%s%s\n", timeStr, "Program Shutdown\n");
    }
    break;
  }    
  g_free(diskMsg);
  return;
}

static void ustreamToEmail(USTREAMS_STREAM stream, const char *msg)                                           
{  // send warning and err messages to one or more email addresses
  gboolean shutDown = USTREAMS_SHUTTING_DOWN(msg);
  gchar *subject, *body;

  switch (shutDown)
  {
    case FALSE:    // construct subject and body of message using the msg being passed in
      subject = g_strdup_printf("%s: Warning/Error", procName);
      body = g_strdup_printf("%s", msg);
    break;
    case TRUE:    // construct subject and body of message based on fact that 
            // the stream is shutting down, for example:
      subject = g_strdup_printf("%s: Shutdown Msg", procName);
      body = g_strdup_printf("Program %s has Completed and is Shutting Down.\n", procName);
    break;
  }

  // send message to all relevant email recipients
  // or perhaps to a twitter account, 
  // or to another program for further processing
  // ...
  // ...

  g_free(subject);
  g_free(body);
  return;
}

// a fake successful connection to a database
// reverse comment these lines for example of simulated successful and failed DB connections
//#define CONNECTTODB FALSE
#define CONNECTTODB TRUE

int main(int argc, char **argv)
{
  procName = g_strdup("ustreams-example");

  fprintf(stdout, "Before initializing ustreams, stdout works as normal...\n");
  fprintf(stdout, "But all messages issued after ustream initialisation will now be intercepted by the ustreams library.\n");
  fprintf(stdout, "Check the files in the logs directory for all further messages.\n");

  // initialize ustreams
  // example init call to *not* re-direct stdout or stderr
  // if (!ustreams_init(USTREAMS_STDNONE, TRUE, TRUE))      
  if (!ustreams_init(USTREAMS_STDIN |  USTREAMS_STDOUT | USTREAMS_STDERR, TRUE, TRUE))
  {
    fprintf(stderr, "Initialisation of U-Stream Library failed, Necessarily Quitting...\n");
    exit(-1);
  }
  fprintf(stdout, "So this message, now, will not be seen in a normal terminal output,\n"
          "\tbut in our disk-file instead.\n");
  fprintf(stdout, "If you desire that stdout should continue sending to the terminal,\n"
          "\tyou have the option to remove it from the list of streams that the\n"
          "\tustreams library will intercept.\n");

  // register all streams for messages to be written to a disk-file
  ustreams_register(USTREAMS_REGISTER, USTREAMS_LOG, ustreamToDisk);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_INFO, ustreamToDisk);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_WARN, ustreamToDisk);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_ERR, ustreamToDisk);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_DBUG, ustreamToDisk);

  // additionally register warning and error streams to be forwarded via email to somewhere relevant
  ustreams_register(USTREAMS_REGISTER, USTREAMS_WARN, ustreamToEmail);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_ERR, ustreamToEmail);

  // All callbacks are now registered, but since we set initial msg cacheing to TRUE, 
  // we won't "see" any messages until we tell them to be forwarded to our callbacks.
  // This is desirable since some programs may want to confirm success of some event before proceeding
  // e.g., successfully connecting to a DB     
  fprintf(stdout, "Attempting connection to Database FOO\n");  
  if (CONNECTTODB == FALSE)
  {
    fprintf(stdout, "Connection to DB FOO Failed!\n");
    fprintf(stderr, "Connection to DB FOO Failed!\n");
    ustreams_cmd(USTREAMS_SHUTDOWN);
    exit(-1);
  }
  fprintf(stdout, "Connection to DB FOO successful.\n");

  // connection to DB is successful, so register all streams for our DB writer callbacks
  ustreams_register(USTREAMS_REGISTER, USTREAMS_LOG, ustreamToDB);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_INFO, ustreamToDB);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_WARN, ustreamToDB);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_ERR, ustreamToDB);
  ustreams_register(USTREAMS_REGISTER, USTREAMS_DBUG, ustreamToDB);

  // convert ustream processing from internal cacheing to sending to our callbacks
  // all messages cached to this point in time are also forwarded to the callbacks immediately 
  ustreams_cmd(USTREAMS_OUTPUT);

  fprintf(stdout, "A stdout message\n");
  fprintf(stderr, "A stderr message\n");
  fprintf(stdinfo, "A stdinfo message\n");
  fprintf(stdwarn, "A stdwarn message\n");
  fprintf(stddbug, "A stddbug message\n");

  // now we de-register the Write-to-Disk callback for all streams
  ustreams_register(USTREAMS_DEREGISTER, USTREAMS_OUT, ustreamToDisk);
  ustreams_register(USTREAMS_DEREGISTER, USTREAMS_INFO, ustreamToDisk);
  ustreams_register(USTREAMS_DEREGISTER, USTREAMS_WARN, ustreamToDisk);
  ustreams_register(USTREAMS_DEREGISTER, USTREAMS_ERR, ustreamToDisk);
  ustreams_register(USTREAMS_DEREGISTER, USTREAMS_DBUG, ustreamToDisk);

  // these messages, then, will only get delivered to the callbacks still registered
  // N.B. all messages sent to a stream having no callback registration are silently droppped
  fprintf(stdout, "A stdout message, no longer delivered to Disk\n");
  fprintf(stderr, "A stderr message, no longer delivered to Disk\n");
  fprintf(stdInfo, "A stdInfo message, no longer delivered to Disk\n");
  fprintf(stdWarn, "A stdWarn message, no longer delivered to Disk\n");
  fprintf(stdDbug, "A stdDbug message, no longer delivered to Disk\n");

  ustreams_cmd(USTREAMS_SHUTDOWN);

  exit(0);
}

Sample Implementation - Output

The Sample Implementation provided with the delivery provides a completed implementation of the sample code above. Output to DB and e-mail is simulated by writing to a disk file.

The complete sample implementation generates the following output results:

::::::::::::::
ustreams-example.DB.dbug
::::::::::::::
INSERT INTO `LOGS` (type, msg) VALUES ('DBUG', 'A stdDbug message, first cached')
INSERT INTO `LOGS` (type, msg) VALUES ('DBUG', 'A stdDbug message, not cached')
INSERT INTO `LOGS` (type, msg) VALUES ('DBUG', 'A stdDbug message, no longer delivered to Disk File')

::::::::::::::
ustreams-example.DB.err
::::::::::::::
INSERT INTO `LOGS` (type, msg) VALUES ('ERR', 'A stderr message, first cached')
INSERT INTO `LOGS` (type, msg) VALUES ('ERR', 'A stderr message, not cached')
INSERT INTO `LOGS` (type, msg) VALUES ('ERR', 'A stderr message, no longer delivered to Disk File')

::::::::::::::
ustreams-example.DB.info
::::::::::::::
INSERT INTO `LOGS` (type, msg) VALUES ('INFO', 'A stdInfo message, first cached')
INSERT INTO `LOGS` (type, msg) VALUES ('INFO', 'A stdInfo message, not cached')
INSERT INTO `LOGS` (type, msg) VALUES ('INFO', 'A stdInfo message, no longer delivered to Disk File')

::::::::::::::
ustreams-example.DB.log
::::::::::::::
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'So this message, now, will not be seen in a normal terminal output,')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', ' but in our disk-file instead.')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'If you desire that stdout or stderr should continue sending to the terminal,')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', ' you have the option to remove one or both from the ustreams_init() command.')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', ' In this case, the U-Streams library will not intercept these streams.')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'A stdout message, first cached')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'Attempting connection to Database FOO')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'Connection to DB FOO successful.')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'A stdout message, not cached')
INSERT INTO `LOGS` (type, msg) VALUES ('LOG', 'A stdout message, no longer delivered to Disk File')

::::::::::::::
ustreams-example.DB.warn
::::::::::::::
INSERT INTO `LOGS` (type, msg) VALUES ('WARN', 'A stdWarn message, first cached')
INSERT INTO `LOGS` (type, msg) VALUES ('WARN', 'A stdWarn message, not cached')
INSERT INTO `LOGS` (type, msg) VALUES ('WARN', 'A stdWarn message, no longer delivered to Disk File')

::::::::::::::
ustreams-example.Disk.dbug
::::::::::::::
2012 203 09:10:26   A stdDbug message, first cached
2012 203 09:10:26   A stdDbug message, not cached

::::::::::::::
ustreams-example.Disk.err
::::::::::::::
2012 203 09:10:26   A stderr message, first cached
2012 203 09:10:26   A stderr message, not cached

::::::::::::::
ustreams-example.Disk.info
::::::::::::::
2012 203 09:10:26   A stdInfo message, first cached
2012 203 09:10:26   A stdInfo message, not cached

::::::::::::::
ustreams-example.Disk.log
::::::::::::::
2012 203 09:10:26   So this message, now, will not be seen in a normal terminal output,
2012 203 09:10:26       but in our disk-file instead.
2012 203 09:10:26   If you desire that stdout or stderr should continue sending to the terminal,
2012 203 09:10:26       you have the option to remove one or both from the ustreams_init() command.
2012 203 09:10:26       In this case, the U-Streams library will not intercept these streams.
2012 203 09:10:26   A stdout message, first cached
2012 203 09:10:26   Attempting connection to Database FOO
2012 203 09:10:26   Connection to DB FOO successful.
2012 203 09:10:26   A stdout message, not cached

::::::::::::::
ustreams-example.Disk.warn
::::::::::::::
2012 203 09:10:26   A stdWarn message, first cached
2012 203 09:10:26   A stdWarn message, not cached

::::::::::::::
ustreams-example.email.err
::::::::::::::
To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, first cached

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, first cached

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, not cached

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, not cached

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, no longer delivered to Disk File

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stderr message, no longer delivered to Disk File

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Shutdown Msg
Message:    Program ustreams-example has Completed and is Shutting Down.


To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Shutdown Msg
Message:    Program ustreams-example has Completed and is Shutting Down.


::::::::::::::
ustreams-example.email.warn
::::::::::::::
To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, first cached

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, first cached

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, not cached

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, not cached

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, no longer delivered to Disk File

To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Warning/Error
Message:    A stdWarn message, no longer delivered to Disk File

To: alerts@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Shutdown Msg
Message:    Program ustreams-example has Completed and is Shutting Down.


To: warnings@somecompany.com
From:   ustreams-example proc
Subject:    ustreams-example: Shutdown Msg
Message:    Program ustreams-example has Completed and is Shutting Down.

Read all 3 comments »

Questions & Comments

You must be logged-in to leave a comment.
Log-in now or register for a free account.

Omar Juarez said:

10 months ago
Hi Richard, is there a way to also do a stdin redirection ? What I need is to make a GUI terminal emulation to run an already built C program that runs on terminal, but I want to make GUI textfield to input terminal commands and retrieve the stdout to another textview.

Reply Flag comment
- Richard Ivor Author said:
  
  10 months ago
  Hi Omar,
  
  I have updated the U-Streams Library functionality to include interception of stdin, where you register your callback(s) in the same way as for the write streams. All incoming messages through stdin, then, are now routed to the registered callback function(s).
  
  As well, I have also included a new function to allow the caller to designate the message delimiter, per stream.
  
  This functionality is available in the current downloadable version.
  
  Reply Flag comment
Paul North said:

10 months ago
Brilliant concept and quality implementation.

Reply Flag comment

Your cart is empty

Single-App Personal $19.99

Single-App Commercial $79.99

Multi-App Commercial $259.99

Developer License $499.99

A C library to intercept and redirect UNIX stdin, stdout & stderr streams to multiple destinations. An unlimited number of user-defined write streams may also be created for multiple destination output and onward processing.

30 minutes

$6000 / 100h *

Description

Sample Implementation - Described

Sample Implementation - Code

Sample Implementation - Output

Questions & Comments

Leave a comment

User Reviews

Binpress

Source-code components

Publishing on Binpress

Your cart is empty

Single-App Personal $19.99

Single-App Commercial $79.99

Multi-App Commercial $259.99

Developer License $499.99

U-Streams

A C library to intercept and redirect UNIX stdin, stdout & stderr streams to multiple destinations. An unlimited number of user-defined write streams may also be created for multiple destination output and onward processing.

Share and win up to $15k in prizes each month!Learn more

30 minutes

$6000 / 100h *

Description

Related components

Sample Implementation - Described

Sample Implementation - Code

Sample Implementation - Output

Questions & Comments

Leave a comment

User Reviews

Binpress

Source-code components

Publishing on Binpress