Thursday, May 26, 2011

Critical system files in Linux

In Linux, almost all configuration parameters are stored in ordinary text files. And there is a special location under which the configuration files are stored namely /etc. The following table lists all major configuration files found in Linux and their purpose.

Critical system files in Linux
File/DirectoryPermissionsDescription
/var/log/751Directory containing all log files.
/var/log/messages644System messages.
/etc/crontab600System wide crontab file.
/etc/syslog.conf640Syslog daemon configuration file.
/etc/logrotate.conf640Controls rotation of system log files.
/var/log/wtmp660Who is logged in now. Use who to view.
/var/log/lastlog640Who has logged in before. Use last to view.
/etc/ftpusers600List of users who cannot FTP to the machine.
/etc/passwd644List of system’s user accounts.
/etc/shadow600Contains encrypted account passwords.
/etc/pam.d750PAM configuration files.
/etc/hosts.allow600Access control file.
/etc/hosts.deny600Access control file.
/boot/grub/grub.conf600Boot configuration file for GRUB bootloader.
/etc/securetty600TTY interfaces that allow root logins.
/etc/shutdown.allow400Users allowed to ctrl-alt-del
/etc/security700System access security policy files.
/etc/rc.d/init.d/750Program startup files on Red Hat systems.
/etc/init.d/750Program startup files on Debian systems.
/etc/sysconfig751System and network config files on Red Hat.
/etc/ssh750Secure shell configuration files.
/etc/sysctl.conf400Contains kernel tunable options.

Tuesday, May 24, 2011

Free Up Cache Memory in Linux

In the past, I've been forced to do ridiculous things like cat a file larger than available RAM to /dev/null and edit gigabyte files which flood my cache with this data. Luckily, Linux kernels 2.6.16 and newer provide a mechanism to clear the inode, page, and dentry caches on demand avoiding all this headache. All you have to do is echo a value to the proc filesystem, and you're done.

To use /proc/sys/vm/drop_caches, just echo a number to it.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation and dirty objects are not freeable, the user should run "sync" first!

This was originally found @ http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

Screen Quick Reference

http://aperiodic.net/screen/quick_reference

Thursday, May 19, 2011

Hadoop Link

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

How To – Find Out Your Ubuntu Version Name

This is just a quick tip, and something I tend to need to do because I’m a bit absentminded and have one of the worst short-term memories out there. The version names for Ubuntu sometimes escape me. Maybe it’s because they are referred to by cute little ‘codenames’ instead of version numbers.

Type the following at the prompt:
cat /etc/lsb-release
It should output something similar to mine, which looks like this:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=Natty Narwhal
DISTRIB_DESCRIPTION="Ubuntu 11.04"

Not too hard to figure out that the line that says ‘DISTRIB_CODENAME’ is the one that tells you the name of your version. Fairly painless.

You can find it by executing the below command

cat /etc/issue
The file /etc/issue holds the version of Ubuntu installed on your system


Wednesday, May 18, 2011

Chapter 8. Configuring Firewalling

8.1 Overview of Firewalling

In this chapter we implement packet filtering to protect our Access Point and our local wired network. It is helpful to think of the wireless network as dirty just as we think of the Internet; We have little to no control over who uses the wireless network and for what purposes, so we do not trust it. Further, the data and systems we wish to protect are on our wired LAN.

The standard tool for packet filtering on modern Linux systems is called IPTABLES. It is a collection of kernel modules and userland tools which can be used to formulate complex firewalling solutions. More in-depth guides to iptables can be obtained from the netfilter homepage, the Linux 2.4 Packet Filtering HOWTO and of course the iptables manpage. IPTABLES is included in both Redhat kernels dealt with during this HOWTO. If you are using a different distribution or kernel that doesn't support IPTABLES, you will need to obtain or build a kernel that does.

Note that the IPTABLES configuration given in this chapter will be superceded by the NOCAT configuration outlined in Appendix B as NOCAT configures IPTABLES on the fly. Readers who intend to runNOCAT for HOTSPOTing on their Access Point can safely skip this section.


8.2 Example IPTABLES configuration file

There are a number of GUI tools for configuring IPTABLES, both free and commercial, but we will cut to the chase and edit the IPTABLES configuration file directly. As in other sections, presented here is an example configuration file which you can modify for your use, using the imbedded comments as a guide. In this case, lines beginning with a "#" are comments.

Note that this is a very simple IPTABLES configuration file. It has no facility for logging, reporting or Network/Port address translation. Outside of the core functionality required to make the various services on our Access Point available, nothing is catered for. Further, it assumes that NO traffic is to be allowed on to the LAN from the WLAN except for responses to LAN generated traffic, which may or may not suit your purposes. In short, you can use this example as the basis of your firewalling policy but will need to add to it or modify it extensively in order to make it suitable for use in your environment.

For Redhat, the IPTABLES configuration file is /etc/sysconfig/iptables - Which is used by the /etc/init.d/iptables startup script.

# Example /etc/sysconfig/iptables configuration file
#
# Turn on traffic filtering
*filter

# Set default policies
:INPUT DROP [1:44]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [27040:2493902]

# Accept all traffic from the loopback interface.
-A INPUT -i lo -j ACCEPT

# Accept legitimate responses to traffic we generate.
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Accept SSH connections from the wired network only.
# Change this to your LANs IP network
-A INPUT -s 192.168.5.0/255.255.255.0 -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s ! 192.168.5.0/255.255.255.0 -p tcp -m tcp --dport 22 -j DROP

# Allow ICMP, though there is a case for disabling it on the WLAN interface.
-A INPUT -s 192.168.5.0/255.255.255.0 -i eth0 -p icmp -j ACCEPT
-A INPUT -s 10.0.0.0/255.0.0.0 -i wlan0 -p icmp -j ACCEPT

# Allow inbound DNS requests from the wireless network.
-A INPUT -i wlan0 -p udp --dport 53 -j ACCEPT
-A INPUT -i wlan0 -p tcp --dport 53 -j ACCEPT

# Allow inbound DNS responses from our ISPs DNS servers.
# Change these to the IP addresses of your ISPs DNS servers.
-A INPUT -s 0.0.0.0 -i eth0 -p udp -m state --state ESTABLISHED -m udp --sport 53 -j ACCEPT
-A INPUT -s 0.0.0.0 -i eth0 -p tcp -m tcp --sport 53 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -s 1.1.1.1 -i eth0 -p udp -m state --state ESTABLISHED -m udp --sport 53 -j ACCEPT
-A INPUT -s 1.1.1.1 -i eth0 -p tcp -m tcp --sport 53 -m state --state ESTABLISHED -j ACCEPT

# Allow inbound DHCP from the Local wireless network (note: not from 10.0.0/8)
# Change this to the network allocated for your use.
-A INPUT -s 10.1.2.0/255.255.255.0 -i wlan0 -p udp --dport 67:68 --sport 67:68 -j ACCEPT

# Allow inbound HTTP from the wireless network. Remove the "#" on the next line to enable.
# -A INPUT -s 10.1.2.0/255.255.255.0 -i wlan0 -p tcp -m tcp --dport 80 -j ACCEPT

# Allow inbound FTP from the entire wireless network. Remove the "#" on the next two lines to enable.
# -A INPUT -d 10.1.2.0/255.255.255.0 -p tcp -m tcp --dport 21 -j ACCEPT
# -A INPUT -d 10.1.2.1 -p udp -m state --state NEW,ESTABLISHED -m udp --dport 21 -j ACCEPT

# Allow all related traffic to/from non-privileged ports.
-A INPUT -p tcp -m tcp --sport 1024:65535 --dport 1024:65535 -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow all traffic from the LAN to be forwarded to the WLAN.
# Change this to your LANs IP network
-A FORWARD -s 192.168.5.0/255.255.255.0 -i eth0 -o wlan0 -d 10.0.0.0/255.0.0.0 -j ACCEPT

# Forward all legitimate responses to forwarded traffic.
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

# Make it all true.
COMMIT
# Completed on Thu Jan 30 18:35:03 2003



8.3 Testing IPTABLES

Now that our configuration file is in place, we can test firewalling. We do this by ensuring that IPTABLES starts correctly, checking that the necessary modules have loaded and testing that connectivity between systems on our networks is filtered as we intended.

8.3.1 Starting IPTABLES manually

We start up IPTABLES with the following command, which will use the /etc/init.d/iptables script to load the necessary kernel modules and set out firewalling rules by loading the entries we made in/etc/sysconfig/iptables in section 8.2 above;

[root@accesspoint root]# service iptables start

Which should produce the following output;

Flushing all current rules and user defined chains: [ OK ]
Clearing all current rules and user defined chains: [ OK ]
Applying iptables firewall rules: [ OK ]

Take note of any errors as they will give us hints as to what has gone wrong. More often than not they will be syntax errors in the configuration file. If not, we can need to check that the necessary modules have loaded.

8.3.2 Checking that the IPTABLES modules have loaded

The kernel needs to load a number of modules in order for the firewalling functionality we require to be supported. We can check that the necessary modules have loaded with the following command;

[root@accesspoint root]# lsmod

Ensuring that the following modules are present. (Note that modules irrelevant to IPTABLES have been omitted from this list)

ipt_REJECT 3992 0 (autoclean)
iptable_filter 2444 1 (autoclean)
ip_tables 15096 3 [ipt_state ipt_REJECT iptable_filter]
ipt_state 1080 8 (autoclean)
ip_conntrack 27272 1 (autoclean) [ipt_state]


If the various IPTABLES modules are not present we can try manually inserting them into the kernel with the following command, again taking note of any errors for clues as to what may have gone wrong;

[root@accesspoint root]# insmod ip_tables


8.3.3 Using the IPTABLES userspace command

In order to see a list of the running IPTABLES rules, insert or delete new ones and generally manage firewalling, we can use the /sbin/iptables command. Refer to the iptables manpage for further information. The following command dumps the current running rules used by the kernel.

[root@accesspoint root]# /sbin/iptables -L

Which should show output similar to the following;

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- 192.168.5.0/24 anywhere tcp dpt:ssh
DROP tcp -- !192.168.1.0/24 anywhere tcp dpt:ssh
ACCEPT icmp -- 192.168.5.0/24 anywhere
ACCEPT icmp -- 10.0.0.0/8 anywhere
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- 0.0.0.0 anywhere state ESTABLISHED udp spt:domain
ACCEPT tcp -- 0.0.0.0 anywhere state ESTABLISHED tcp spt:domain
ACCEPT udp -- 1.1.1.1 anywhere state ESTABLISHED udp spt:domain
ACCEPT tcp -- 1.1.1.1 anywhere state ESTABLISHED tcp spt:domain
ACCEPT udp -- 10.1.2.0/24 anywhere udp spts:bootps:bootpc dpts:bootps:bootpc
ACCEPT tcp -- anywhere anywhere state RELATED,ESTABLISHED tcp spts:1024:65535 dpts:1024:65535

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- 192.168.5.0/24 10.0.0.0/8
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Note that the information displayed is how IPTABLES has translated our configuration file. This can at times be misleading as it leaves some details out. If in doubt, refer to the configuration file.


8.3.4 Other methods of testing

Further testing of firewalling can be achieved by testing connectivity from other systems on your wired and wireless networks. The NMAP port scanner is invaluable in this regard and I recommend that you use it, both from your Access Point and from systems on your networks. If you did not install NMAP during our initial Redhat installation in Chapter 3, you can download and install it.

Further testing of your firewalling setup can be achieved simply by attempting to ping between systems on your networks, or attempting to connect to services that you have running. For instance, you may like to check that wireless clients obtain IP configuration successfully from the Access Point via DHCP and that they can perform DNS lookups. You may also like to confirm that wireless clients cannot ping machines on your wired LAN but that machines on your LAN can ping wireless clients.

Security is always a major concern and it is over to you to audit your network and ensure that it is adequately protected on an on-going basis.


8.4 Enabling IPTABLES from startup

As we did with DHCPD, NAMED, ZEBRA and OSPFD in previous chapters, we need to turn on IPTABLES from boot using the setup utility. We add IPTABLES to our list of services that should start at boot time by adding an asterix beside the IPTABLES entry in the System services menu of setup as described in section 3.3

An Introduction to memcached

"Caching" is a term you've probably heard mentioned before in various places (including this site). The idea behind caching is to store a copy of some piece of data so you can re-use it again later without jumping through whatever hoops you had to go through the first time to get it. There are different ways you can cache data (queries, objects, etc) and different medium in which you can store the cache (files, database, memory). Any way you do it, the main goal of caching is to increase the performance of your site or application. In many cases caching is used to lessen the amount of interaction with the database, which increases performance and decreases the load on your server.

I would like to talk about my personal favorite method of caching: memcached. I'll show you how memcached works, how to install it, and how to use it to help your site/application run faster and scale better. According to the memcached site, "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load." In plain English, this means memcached is an application that you can use to take advantage of spare free memory on any number of machines to cache pretty much anything you want (with a few exceptions) and retrieve it very quickly. Memcached was originally developed by Danga Interactive to help speed up LiveJournal. Some of memcached's great features are that in runs on a number of platforms (Linux, BSD, Windows), is VERY fast, and has a number of client APIs already written so you'll more than likely find libraries for any type of project you're working on. We'll focus on the PHP API in this article.


Before I get too far, I want to mention a couple of alternatives that may fit your particular situation.

Local Database Query Cache: Your database may have it's own native query caching, which you don't have to do much to use. The only drawback is that if a table is updated, its entire cache is thrown out.

The PHP APC extension: The APC extension is an opcode cache for your PHP scripts, but also provides a similar function to that of memcache. The biggest problem with APC is that you can only access the local APC cache. There are other distributed caching systems, such a MCache, but I have no personal experience with any of these, so I cannot opine on any advantages or disadvantages of using another tool.

Installation

First thing we need is an instance of memcached to store our data in. Unix/Linux folks can download the source from here and follow the instructions for installation on this page. Some distributions (like Ubuntu and CentOS) have memcached in their repositories, so you can use the native package installer like apt or yum to install memcache. Windows users can find binaries and installation instructions at http://jehiah.cz/projects/memcached-win32/. Any other questions you may have about memcached can possibly be answered on their excellent FAQ.

Now that we have memcached up and running, we need a way to talk to it. This is where the client APIs come in. We'll be using the PHP API, so I'll show you how to install the PHP memcached extenstion. The easiest way to do this is using the 'pecl' command: [pecl install memcache] this will download, compile, and install the extension. All you will need to do is add the line 'extension=memcache.so' to your php.ini file. You may also need to find and move the memcache.so file into the extensions/ directory (usually located at /usr/local/lib/php/extensions) by hand. Other options for installing PHP extensions and instructions for windows users can be found at http://us2.php.net/manual/en/install.pecl.php.

Implementation

Now that all the pieces are in place, let's integrate memcached into our application. First thing we need to do is to connect to our memcached server:

PLAIN TEXT
PHP:

$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");
?>


This is assuming that memcached is running on the local machine and it's using the default settings. You would usually do this connection when you open a database connection at the beginning of your application. If you want to connect to more than one memcached server, simply call $memcache->connect() again and pass in the name and port number of the additional server(s). Now that we've got a connection, let's look at this section of code:

PLAIN TEXT
PHP:

$sql = "select * from pages where page_id=1";
$qry = mysql_query($sql) or die(mysql_error()." : $sql");
$result = mysql_fetch_object($qry);
$content = $result->content;
?>


This fetches the 'content' field from our pages table. Now, if the data in the content field does not change very often, it is a good candidate for caching. Here's one possible way how we would integrate memcached into our little section of code:

PLAIN TEXT
PHP:

//write query
$sql = "select * from pages where page_id=1";
//create an index key for memcache
$key = md5('query'.$sql);
//lookup value in memcache
$result = $memcache->get($key);
//check if we got something back
if($result == null) {
//fetch from database
$qry = mysql_query($sql) or die(mysql_error()." : $sql");
if(mysql_num_rows($qry)> 0) {
$result = mysql_fetch_object($qry);
//store in memcache
$memcache->set($key,$result,0,3600);
}
}
$content = $result->content;
?>


A bit more involved, but we are now using memcached! The above code first checks to see if we can find whatever it is we are looking for in memcache, and if we can't find it, we fetch it from the database and use the result to populate the cache. In this example, I stored the entire $result object in cache and set its expiration to 3600 seconds (1 hour). The third flag in the set() function deals with whether to compress the data or not. Depending on your needs, you can store strings, numbers, objects, and arrays in memcache. Anything that is serializable in PHP can be cached, so database connections and file handles won't work.

Now that we're pulling data from memcache, what happens if the data in the database is updated? We can compensate for this in two ways. The easiest is to pass an expiration on the data that is fairly low, but you'll have to deal with a little lag from the time you updated the database to when it will appear in the cache. The other way is to update the cache on the fly any time an update or delete occurs. This involves a bit more work as you may have to update many places in the cache depending on how many queries could possibly touch the data, but this is only necessary when doing query caching as in my example rather than just straight content caching.

Memcached affords us endless possibilities (query caching, content caching, session storage) and great flexibility. It's an excellent option for increasing performance and scalability on any website without requiring a lot of additional resources.