How to get log data of huge file for certain timestamp

Hey Friends,

All the system Admin for any application supported by Unix / Linux may face this common issue multiple times, when he has to look for the root cause for any issue encountered. We know the log file helps alot in identifying the root cause of any issue, but we know its not that easy, even if you filter or grep with some special character strings. But the best part is, that log files are generated with respect to timestamp (as in my case its yyyy-mm-dd HH:MM:SS format eg. for 1-Oct-2016 00 hours it will be as: 2016-10-01 00:00:00)

There may also be requirement when you need to get all data for certain period of file.

Now let us focus on what commands can help us on such requirements based on case:

Case-1 - A User needs issue.out file data from 1st October to 5 October
Solution -
You need to execute command to filter required data & then redirect that data to other file

Command-

grep "2016-10-0[1-5]" issue.out > required_data.out Case-2 - A User needs something.log file but only for error data of 1st October -2016 from 10 am to 11 am server time Solution - You need to execute command to filter required data & then redirect that data to other file (required_data.out in my case) Command- grep "2016-10-01 1[0-1] " something.log | grep -i 'error' > required_data.out These commands will help you to dig out the main cause of any issue instantly... :) Note - Redirection is done 2 ways either by > ( to clean older date & make fresh data from first line) and >> which will append data from last line in same file.

Flushing File every n minutes in Unix / Linux

Hey Friends,

We all know that best way to handle important log files is to set log file rotation policy, but if accidentally any log file is generating & no rotation policy is set. Then this file may hamper disk-space, which may result in fail-over of applications within the mount-point.
So we have a good provision to schedule cronjob based on time interval to flush any such files.

First let us focus on command required to flush data in file:
Let us assume file is under /tmp mountpoint with name as: heavy.log

Command to Flush -
> /tmp/heavy.log

This will clear data in file. Now let us schedule this command job every 5 mins.

Prerequisite: There must be crontab enabled for owner to schedule the job.

How to edit & update cronjob:
Command -

crontab -e

[This will open cron in edit mode] Now you need to type command in time duration format like below:

* * * * * echo '' > /tmp/heavy.log

This will clear / flush file data every minute, but our requirement is to flush it every n mins. So we modify above command as:
*/n * * * * echo '' > /tmp/heavy.log

Where n is any positive integer value with range (0<n<60)

Now what 5 stars indicate in cron file?
Answer - Below is sequence of every star:
* – Every minute
* – Every hour
* – Every day
* – Every month
* – Every day of the week

So */n will make job schedule every n mins & flush data in file.

Once above job is created, you may exit the edit mode of cron by command as:

:wq!

To verify your cron file post changes: You may type:
crontab -l

[Note - Options to work in crontab are same as vi editor]

Killing Multiple Process with one Command

Hi Friends,

You may face the requirement or concern, when you will need to kill many similar process with one command to save your time.
There is the command you may use for such purpose:

kill -9 `ps -ef | grep <Process_Name> | grep -v grep | awk '{print $2}'`

Note -
<Process_Name> : You need to change with the process you need to kill, having multiple pids.

Copy / Move / Remove (delete) Unix files with specific days

As Unix Server Admins, we keep on facing requirements or issues where we are required to copy / move / remove files of specific date or certain days older. Here i am posting the command & arguments that can be used for such requirements.


Condition - For files from source server to Destination server x days older with same timestamp.

Copying Files

find /source/Directory -mtime x -exec cp -p "{}" /destination/directory/ \;

Moving Files

find /source/Directory -mtime x -exec mv "{}" /destination/directory/ \;

Removing / Deleting Files

find /source/Directory -mtime x -exec rm "{}" \;

Where
mtime - Modified Time
exec - Execute

To Access Unix / Linux / Solaris Server through VNC For Graphical Usage or Installation

Step-1 – Execute Certain VNC Commands to gain access through VNC.

1.      vncserver  [ To gain access with specific port ] 
2.      vncpasswd [ To set password to access server through VNC ]

Step-2 – Click VNC.exe file & you will be prompted to enter server_name:port ( Mention full server name with port)

You can now access Server through VNC.

Finding Files / Directories in Unix / Linux

There is always need to search files or specific 'text_string' in any file in the applications with unix / linux OS servers. Here i am specifying certain commands which will always help you for such requirements and make your task easier.

• To Search Files : find . -type f –name ‘File_Name.ext’

• To Search Directory : find . -type d -name "Folder_Name";



• To Search String Character within File :
grep –i “TEXT_STRING” ./File_Name
or
cat /path/to/file | grep -i 'TEXT_STRING'

• String Search through VI Editor :
vim / vi File_Name ( Enter ) Then
/TEXT_STRING ( Forward slash {/} then the string to search, type n for next occurrence of string & shift+n for previous occurrence )

• Find & Replace String Character in File:
vi File_Name ( Enter ) Then :
%s/Old_String/New_String/g

• To Search all Files with path with specific String:
find . -type f -exec grep -il "String_Text" {} \;       

• To Find files of certain types containing specific 'String_Text':
find . -type f -name "*.*" -exec grep -il "String_Text" {} \;  

[Note - Use File name or *.File_Extension in place of *.*]

• To Find & list out files of UID - appadmin having size > 20MB which are five days older.

find . -user appadmin -type f -size +20M -mtime +5 -exec ls -lrt {} \;

• To Read Specific Line of file:

sed -n '100p' File_Name

Where - 100 is line number for file : File_Name

Note - In Above commands:
# i for Ignore case
# s for search
# g for globally
# f for File
# d for directory / folder
# exec - for Execution
# {} for File-name Placeholder
# \; To end & execute the command

Additional Information :
For executing text make use as below:

• 'Text' => For text without blank space or single word use Single Quote - '
• "Text String" => For text with spaces in between or group of words use double quote - "