FTP Using Perl, File Upload Tool

Topic:  Upload a file via FTP , using Perl Object Oriented Programming  . Check the uploaded file size upon successful in remote host.

Pre-request:  Check Net::FTP package in your system, before use this ftp file upload tool.

Execute this linux command:
$perldoc -l Net::FTP
/usr/lib/perl5/5.8.8/Net/FTP.pm
$

It means already installed . Other do install it from RPM, or CPAN library.

After that you can copy the files below and enjoy make use of reliable FTP File Upload Tool.

Files :

1. ftp. pm  -  <perl ftp user defined class to upload file >
#*******************************************************
#Class Name: ftp                                                                               
#Member Description:    
#        host            -> Remote server IP Addr / Host Name
#        username    -> user name of remote server
#        password    -> password of remote server
#        destpath    -> destination path on remote server
#        binary        -> either transfer in binary mode
#        timout        -> ftp time out value
#        debug            -> ftp with debug log
#        logfile        -> log file name to write ftp logs
#        filefd        -> file descriptor of log file
#********************************************************

2. upload.pl  - <upload a file to remote host using ftp class>

Copy both these files in a single directory.  Change the ftp class params in upload.pl file with respect to need.

Fileame : ftp.pm
-----------------
#---------------------------------------------------
#Class Name: ftp
#Scope         : file upload using ftp protocol
##---------------------------------------------------

use strict;
use warnings;

use Net::FTP;

package    ftp;

#*******************************************************
#Class Name: ftp                                                                               
#Member Description:    
#        host            -> Remote server IP Addr / Host Name
#        username    -> user name of remote server
#        password    -> password of remote server
#        destpath    -> destination path on remote server
#        binary        -> either transfer in binary mode
#        timout        -> ftp time out value
#        debug            -> ftp with debug log
#        logfile        -> log file name to write ftp logs
#        filefd        -> file descriptor of log file
#********************************************************



#-----------------------------------------------------------
#@brief: constructor
#
#@param in: name of the class, required memebers of class
#
#@return value: object of the class
#                       
##-----------------------------------------------------------

sub    new
{
    my    $class    = shift;
    my    $self        =    {@_};

    bless($self, $class);

    $self->{filefd} = undef;

    if( defined($self->{logfile}))
    {
            open($self->{filefd}, ">>$self->{logfile}" ) || die "couldn't open file $!";
    }
   
    return    $self;
}

#-----------------------------------------------------------
#@brief: Class Method for connect and upload the file
#                 to specified destination path at remote server
#
#@param in: object of class, source filaname to be upload
#
#@return value:
#                    size of the file uploaded - on success
#                    -ve - for failure
#-----------------------------------------------------------

sub upload
{
           
    my    ($myclass, $filename) = @_;
       
    my    @date     = ();    
    my    $errno    = undef;
    my    $ret        = undef;

    @date = `date "+%F %H:%M:%S"`;
    chomp(@date);
    printf({$myclass->{filefd}} "%s :: connecting to host [%s]\n",
                 @date, $myclass->{host}) if defined($myclass->{filefd});

    my    $ftp = Net::FTP->new($myclass->{host}, Debug => $myclass->{debug}, Timeout => $myclass->{timeout});
    if ( !defined ($ftp) )
    {
            $errno = -101;
            printf({$myclass->{filefd}} "%s :: connect to host[%s] failed error[%d][%s]\n",
                    @date, $myclass->{host}, $errno, $!) if defined($myclass->{filefd});
          return $errno;
    }

    @date = `date "+%F %H:%M:%S"`;
    chomp(@date);
    printf({$myclass->{filefd}} "%s :: successfully connected to host [%s]\n",
                 @date, $myclass->{host}) if defined($myclass->{filefd});

    if ( $ftp->login($myclass->{username},$myclass->{password}) == 0 )
    {
            $errno = -102;
            printf({$myclass->{filefd}} "%s :: login failed username[%s] password[%s] error[%d][%s]\n",
                @date, $myclass->{username}, $myclass->{password}, $errno, $!) if defined($myclass->{filefd});
            return $errno;
    }
   
    @date = `date "+%F %H:%M:%S"`;
    chomp(@date);
    printf({$myclass->{filefd}} "%s :: login success for username[%s]\n",
            @date, $myclass->{username},) if defined($myclass->{filefd});

    if ( $ftp->cwd($myclass->{destpath}) == 0 )
    {
            $errno = -103;
            printf({$myclass->{filefd}} "%s :: couldn't change to destination directory[%s] error[%d][%s]\n",
                        @date, $myclass->{destpath}, $errno, $!) if defined($myclass->{filefd});
            return $errno;
    }

    if ( $myclass->{binary} == 1 )
    {
                    #$ftp->binary || die $ftp->message;

                    $ret = $ftp->binary;
                    if( !defined($ret) )
                    {
                        $errno = -104;
                        printf({$myclass->{filefd}} "%s :: setting binary mode failed to upload file[%s] error[%d][%s]\n",
                                @date, $filename, $errno, $!) if defined($myclass->{filefd});
                        return $errno;
                    }

                    @date = `date "+%F %H:%M:%S"`;
                    chomp(@date);
                    printf({$myclass->{filefd}} "%s :: uploading file[%s] in binary mode\n",
                            @date, $filename) if defined($myclass->{filefd});
    }
    else
    {
                    @date = `date "+%F %H:%M:%S"`;
                    chomp(@date);
                    printf({$myclass->{filefd}} "%s :: uploading file[%s] in ascii mode\n",
                            @date,    $filename) if defined($myclass->{filefd});
    }

    @date = `date "+%F %H:%M:%S"`;
    chomp(@date);

    $ret = $ftp->put("$filename");
    if ( !defined ($ret))
    {
            $errno = -105;           
            printf({$myclass->{filefd}} "%s :: failed to upload file[%s] error[%d][%s]\n",
                        @date, $filename, $errno, $!) if defined($myclass->{filefd});
            return $errno;
    }

    $ret = $ftp->size($filename);

    printf({$myclass->{filefd}} "%s :: file[%s] upload success size[%d]\n",
                    @date, $filename, $ret) if defined($myclass->{filefd});
    $ftp->quit;

    printf({$myclass->{filefd}} "-" x 100,"\n\n")  if defined($myclass->{filefd});
    printf({$myclass->{filefd}} "\n")  if defined($myclass->{filefd});

    close($myclass->{filefd}) if defined($myclass->{filefd});

    return $ret;       
}

1;


File Name:   upload.pl
-------------------------
#!/usr/bin/perl

#------------------------------------------------------
#Scope: use ftp class & upload the file
#                 to specified destination path on remote server
##------------------------------------------------------

use warnings;
use    strict;

use    ftp;   

my    $sourcefile = "hello.txt";

#initialize the object
my    $myobj = ftp->new(host => "12.1.1.1", username => "root", password => "root123",
                                binary => 1, timout => 120,
                                debug => 0, logfile => "/tmp/ftp.log", destpath => "/tmp");


my $filesize = -s $sourcefile;
my $uploadlen = $myobj->upload($sourcefile);
if($filesize == $uploadlen )
{
    print "ftp upload file[$sourcefile] size[$uploadlen] success \n";
}
else
{
    print "ftp upload file[$sourcefile] size[$uploadlen] failed \n";
}

Fast Traffic Packet Processing In Multi Threaded Environment

Topic:  Discussion on do we require multi thread in network packet processing engines (A packet processing linux based application) 

What is most important to us 

a)      Packet processing must be faster 

                                 i.            There shouldn't be packet queuing in the application

                                ii.            Should use less resource  (i.e)   minimum memory  on the socket receive front

Hence will have less kernel memory utilized

                              iii.            The CPU utilization should not be high ,  overall load average of the CPU should be in control

b)      Database update upon packet processing should be success

                                 i.            There shouldn't be data update missing after packet processing 

                                ii.            Updating your data from the packet shouldn't be delayed  nor missed

How To Approach :

There are multiple architectures which could help to achieve this "Fast Packet Processing Engine".

1)      The first item is all about , what is the data incoming on the receive socket , and what are we doing to do with that data. Do we store it independent  [OR] we just need to act as Gateway [OR]  Relay the packet , and so on .

2)      Based on the data structure design ,  we would able to conclude the architecture for the "Fast Packet Processing Engine"

3)      After deciding the available factors,  we shall go for the "Hardware and CPU Optimized Configurations"

Here Am going to elaborate on  "Fast Packet Processing Engine  For Internet Users"

     

Fast Packet Processing  Engine Specifications:

ü  Multi Processing

ü  Multi Threading  Queue

ü  Radix Tree Implementation for data store

Work Flow :

v    Multiple instance of child processes  -- Decide based on the network traffic in the systems .   A process does packet  receiving functionality and given to  thread worker queue.

v  Pthread worker queue, does the job of handling the packet , and update the data on Radix Databases.  The lock mechanism applied using mutex.

v  The processes scheduling , and load sharing the incoming traffic are implicit from the features of Linux Kernel. Hence the traffic  load distributed  among the number of the  "Child Process Instances"

v  Consider  "Internet User ID" - Example: Mobile Number, Wifi  Client ID, etc . (i.e) A Unique Client ID will be the key for Radix Database.

Achievement :

·          Packet processed at the high rate that of incoming packets.

·         Never packet data got missed, since the pthead packet processing queue will take care of queuing it , if the number of packets are more than the threads.

·         Thereby Fast Packet Processing Engine shall be achieved

Disadvantages:

Ø   Mutex Locking, could lead to race condition,  however as we have handled properly in the Code Implementation, this shall be avoided

Linux Server Specifications:

       Multi Core, Multi CPU Platform , 64 GB RAM (any high end server configuration shall be applied).

FEEL FREE TO COMMENT . 

 Thanks . 

What is advantage of Redis Data Structured Server

Topic : 

Redis data structured server  and sample code , use cases . 

Following essential points , why do we need Redis 

-- Ease of store  and cache key - value pairs 

-- Inbuilt use of  in memory (i.e)  Redis use  memory mapped techniques

-- The key - value cached  values never get lost , as Redis used MMAP concepts 

-- REDIS runs over TCP, UDP , UNIX Domain socket 

-- You can configure Redis either in your local host , or in remote server 

-- Hence Redis shall be called as  Centralized Remote Data Structured Server 

--   Redis has different data types , and has DATABASE similar to other SQL servers 

-- you can create your own data base and have key value mapping stores per data base ... 

--- The data base can be switched to other 

-- Very simple way of doing  SET, GET method to read and write 'key - value ' into redis database server 

-- You can configure redis configuration file , for optimized memory and data base configurations 

Applications  use case of interest 

--- To Maintain centralized unique key value pair

--- Dynamic way of generating unique key and store  value (as input from user) 

Example: 

 You have some data to be stored uniquely across your networks. 

Redis can create you unique key, you shall store your data with that unique key . 

Typically  Redis shall be applied on  User Sessions Accounting,  File Management,  Choosing File with suffix as key, 

Web Caching Contents (file naming as unique key). 

FEEL FREE TO COMMENT FOR CLARIFICATIONS.

 

NetBSD DHCP Client Optimization to discover interfaces

Hi , 

Here is some information about  Internet Systems Consortium (ISF) dhcp client software

ISF DHCP Client software has dhcp client statemachine , well written to acquire ip address for multiple interfaces. 

It maintains three different interface  list for maintain the interface where dhcp client enabled . 

The interface database (list) are 

(a) interfaces 

(b) dummy_interfaces 

(c) fallback_interfaces

Here my share on optimization of discover_interfaces function.

This topic would be used those who are using ISF dhcp client library,  to achieve ip address assignment via dhcp 

discover_interfaces 

-- This function mainly to walk through the interfaces database and get the interfaces who are looking for ip address 

-- In the whole library discover_interfaces function , called for multiple times with different state 

DISCOVER

REQUESTED 

UNCONFIGURED  and so on..

-- I have removed many of the calling places 

-- And introduced passing the interface index as input to discover_interfaces function

--- By this way  i achieved following 

 (a) For every state transition you dont need  to call discover_interfaces 

(b)  During initialization also no need to call discover_interfaces 

(c) The effective use passing interface index solved looking at interface details from the database 

   Instead of walking though the entire interfaces in the system and find out what are interfaces looking for ip address 

--- Two things are achieved ,

1. Calling many times discover_interfaces  (optimized passing interface index and do discover only for the given interface )

2.  Achieved  less CPU Memory Utilization.. 

By this optimization i could able to have more number of interfaces shall able to  get ip address using ISF DHCP Client library. 

IF YOU HAVE ANY COMMENTS ARE CLARIFICATION NEEDED PLEASE FEEL FREE TO COMMENT 

Optimized Receive Buffer on socket descriptor while select socket operation

The optimized way of receiving the data on the socket while doing select.

Scenario :


-- Do select on  "N" number of file descriptors,  based on the FD_ISSET do receive the packet buffer.

-- what happens when there are packets are in stream, (i.e) they are just yet to be copied to socket receive buffer by linux kernel.

-- Obviously after processing the packet, the control go back to select operation


-- Thus having more packets written to the  socket receive buffer during burst scenario, the socket buffer will get full and packets will be lost


-- The optimized approach is to, do  IOCTL call to that socket fd, and get the available receive buffer length  before doing receive packet buffer.

-- Its very optimistic way recently i have experienced this approach..


-- It works pretty cool..

--  you would need to add your code in kernel  soo_ioctl function.. get the struct socket  so_rcv.sb_cc -- i.e length of socket receive buffer

Debugging corrupted stack

Hi,

recently i have experienced having corrupted stack frames.

Its multi threaded environment..

My process got crashed due to stack overflow.


You have memset call where the size you are not initialized .

example:

int n;
char buf[100];

memset(buf, 0, n) ;

(gdb) bt
#0 0x0183e4c8 in byte32sse2 ()
#1 0xb1036f68 in ?? ()
#2 0x00001b98 in ?? ()
#3 0x0183ba51 in _intel_fast_memset ()
#4 0xb1036e10 in ?? ()
#5 0x00000000 in ?? ()
(gdb)    


this is how the intel memset thread stack trace shows. 


It would be difficult to find out, in a process where do you missed to initialize the size n. 


The best approach. 

Run gdb  check for 

 esp  -- top of stack pointer. 

use objectdump -sSLdx  <process binary> 


save the output to a file 

run gdb check for esp 

x/32gwa $esp 

it would possible give the some traces. 

go over the address in the esp 

esp is the top of the stack frame.  

search the address in the  output of objectdump utility. You shall get some clue on  Root Caused.