eucalyptus-selinux: security as a first-class citizen!

Eucalyptus 4.3 development sprint is almost over. SELinux support for Eucalyptus is one of the most exciting features [EUCA-1620] for this release.

Like me, if you haven’t looked into SELinux in a while or it is a new thing to you, here are few tips and tricks that may come handy if things don’t work as expected with SELinux while you are playing with Eucalyptus nightly builds or source code during dev cycles or any other new software while SELinux is set to enforcing on the system.

[It is recommended that you try Eucalyptus 4.3 with SELinux on RHEL 7.x or CentOS 7.x.]

Recently while trying Eucalyptus when SELinux is enabled, I was having issues with starting eucanetd.service, which gave me a chance to look further into Eucalyptus SELinux. It appears that the very latest eucanetd requires to open an UDP port and since SELinux doesn’t yet know about it, it is giving an AVC (Access Vector Cache) error.

In case of situation like this or when a newly installed application fails to run/execute on an SELinux enable box, the first thing to check would be /var/log/audit/audit.log.

In the log I found the following error:

type=AVC msg=audit(1462905888.565:1432): avc: denied { name_bind } for pid=10725 comm="eucanetd" src=63822 
scontext=system_u:system_r:eucalyptus_eucanetd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket

type=SYSCALL msg=audit(1462905888.565:1432): arch=c000003e syscall=49 success=no exit=-13 a0=3 a1=7ffef025a010 a2=10 
a3=7ffef0259d90 items=0 ppid=1 pid=10725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) 
ses=4294967295 comm="eucanetd" exe="/usr/sbin/eucanetd" subj=system_u:system_r:eucalyptus_eucanetd_t:s0 key=(null)

To debug further and start eucanetd.service, I have installed the following tool,
yum install policycoreutils-python

This tools comes with a fantastic command called ‘audit2allow‘, which is useful for both debugging and fixing SELinux policy related issues.

The following command shows the list of SELinux failures with reasons.

audit2allow --why --all

Example:

type=AVC msg=audit(1462905222.124:825): avc:  denied  { name_bind } for  pid=2304 comm="eucanetd" src=63822 scontext=system_u:system_r:eucalyptus_eucanetd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket
	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1462905888.565:1432): avc:  denied  { name_bind } for  pid=10725 comm="eucanetd" src=63822 scontext=system_u:system_r:eucalyptus_eucanetd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket
	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

Update:
Though the above information is okay, but if you are submitting an issue for a project or asking for SELinux related queries, ausearch is probably a better tool as it combines the SYSCALL and AVC records. Thanks Garrett Holmstrom for the suggestion.

Example:

ausearch -c eucanetd --start recent

Output:

----
time->Tue May 10 17:05:36 2016
type=SYSCALL msg=audit(1462925136.650:5237): arch=c000003e syscall=49 success=no exit=-13 a0=3 a1=7ffe13b6d920 a2=10 
a3=7ffe13b6d6a0 items=0 ppid=1 pid=6069 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) 
ses=4294967295 comm="eucanetd" exe="/usr/sbin/eucanetd" subj=system_u:system_r:eucalyptus_eucanetd_t:s0 key=(null)

type=AVC msg=audit(1462925136.650:5237): avc:  denied  { name_bind } for  pid=6069 comm="eucanetd" src=63822 
scontext=system_u:system_r:eucalyptus_eucanetd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket

We can see that eucalyptus_eucanetd type (eucalyptus_eucanetd_t) is missing rules for access. In my case I had quite a few, but the above example gives an idea of how it might look.

Newbie tips:
Run the following command to check security context of a file on a SELinux enabled system,

# ls -ltrhZ /usr/sbin/eucanetd

-rwxr-xr-x. root root system_u:object_r:eucalyptus_eucanetd_exec_t:s0 /usr/sbin/eucanetd

selinux-penguin-new_medium

Now that we’ve identified the problem, solving it is easy. We can use audit2allow to generate modules to install necessary policies. Though, this can be a tedious job sometimes.

To generate missing policies, we can run the following command,

audit2allow -M my_eucanetd_policy < /var/log/audit/audit.log

which will generate an output like below:

******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i my_eucanetd_policy.pp

Now load the module as it says in the output.

At this point we try to run the service again, if it’s a new service, it may fails due to not enough permission, e.g a typical service may require following access for a file { getattr read open } in multiple system calls, but generate policy with audit2allow may only detect the last error.

So, if the service fails to start, we check the audit.log, if it’s another AVC denial error, run the command again to generate module, continue this process until there is no AVC denial in the audit.log.

By this time there could be a couple of policy modules generated and we may want have one single policy file to make future usage easier. After merging the policy files, we need to compile and create policy module package and then install module package like above.

For example after merging policies, mine looked like following:
[WARNING]
this post is for debugging and learning purposes only, under no circumstance you should need to apply custom policies for Eucalyptus. Please open an issue if you run across an AVC denial.


module eucanetd_policy 1.0;

require {
        type user_tmp_t;
        type unreserved_port_t;
        type eucalyptus_cloud_t;
        type eucalyptus_eucanetd_t;
        type eucalyptus_var_lib_t;
        type node_t;
        type eucalyptus_var_run_t;
        class capability { dac_read_search dac_override };
        class file { create getattr write open };
        class udp_socket { name_bind node_bind };
        class dir { read write search add_name };
}

#============= eucalyptus_cloud_t ==============
allow eucalyptus_cloud_t self:capability { dac_read_search dac_override };
allow eucalyptus_cloud_t user_tmp_t:dir read;

#============= eucalyptus_eucanetd_t ==============
allow eucalyptus_eucanetd_t eucalyptus_var_lib_t:dir search;
allow eucalyptus_eucanetd_t eucalyptus_var_run_t:dir { write add_name };
allow eucalyptus_eucanetd_t eucalyptus_var_run_t:file { create getattr write open };

allow eucalyptus_eucanetd_t node_t:udp_socket node_bind;
allow eucalyptus_eucanetd_t unreserved_port_t:udp_socket name_bind;

Update:
If planning to merge to another selinux module, the following command could be provide outputs precisely, thanks Tony Beckham for the suggestion.

ausearch --start recent --success no | audit2allow

After merging all the policies generated by audit2allow, it needs to be compiled and then packaged to be installed as a module.

# compile policy
checkmodule -M -m eucanetd_policy.te -o eucanetd_policy.mod

# create module package
semodule_package -m eucanetd_policy.mod -o eucanetd_policy.pp

# install module
semodule -i eucanetd_policy.pp

By following the above steps, typically we should be able to avoid problems like AVC denials.

The source for eucalyptus-selinux: https://github.com/eucalyptus/eucalyptus-selinux
Report for Eucalyptus related issues: https://eucalyptus.atlassian.net

dhcp-common conflicts with file from package dhclient

I was downloading bunch of dependencies on CentOS 6.0_x86_64. Where I found this silly error.

Transaction Check Error:
file /usr/share/man/man5/dhcp-eval.5.gz from install of
dhcp-common-12:4.1.1-25.P1.el6_2.1.x86_64 conflicts with
file from package dhclient-12:4.1.1-12.P1.el6.x86_64

file /usr/share/man/man5/dhcp-options.5.gz from install of
dhcp-common-12:4.1.1-25.P1.el6_2.1.x86_64 conflicts with
file from package dhclient-12:4.1.1-12.P1.el6.x86_64

But the solution was really easy.

yum update dhclient

NTP server how to

 
Network Time Protocol (NTP)
It’s a protocol used to synchronize Linux system’s clock with an accurate time source. This little yet powerful tool is very important for every big network. And I have few bad experiences with this NTP. Mostly because of either I forgot to configure or ignored for the time being and forgot later.

As I am working with Eucalyptus cloud platform, NTP is a must for the system to work perfectly. It is also used in Openstack cloud platform. In Eucalyptus if you ever find that Cloud Controller and Node Controller is not talking or Node controller is not sharing resources and throwing an error it is very wise to check the NTP configuration in the first place, and make sure that all the machines are following the correct time protocol.

[EUCAERROR ] ERROR: DescribeResource() could not be invoked (check NC host, port, and credentials)

Well, for both the platforms, there are two basic installation needed, one as a server and others as clients. Commands are mostly same for all the Linux platforms.

First install NTP on the server.

$ yum install ntp

it is important that we make sure that run level is set properly

$ chkconfig ntpd on

on server lets synchronize the system clock with an outer source which is more reliable.

$ ntpdate pool.ntp.org

/etc/ntp.conf
for advanced configuration, we can set multiple sources here. Here is a very important catch. Servers who are able to figure out what time it is all on their own, without using the Internet are stratum 1 and then secondary stratum 2. for example,

ntp.amnic.net # stratum 1 timeserver
ntp.adc.am    # stratum 2 timeserver

Then if we want to restrict the access of the servers on our machine,

restrict ntp.amnic.net   mask 255.255.255.255 nomodify notrap noquery
restrict ntp.adc.am      mask 255.255.255.255 nomodify notrap noquery

noquery denies ntpq and ntpdc queries. So the time service does not get affected.
nomodify denies ntpq and ntpdc queries which attempt to modify the state of the server.
Instead of noquery, notrap will still allow queries from ntpq and ntpdc. traps are used by remote event logging programs. Traps provides a way off collecting ntpd information from another machine and require the use a special trap client program.

When the server is going to provide time to a certain network, just allow the network to query the NTP server

restrict 10.10.10.0 mask 255.255.255.0 nomodify notrap

The localhost needs to have the full access, so make sure it has no restriction keywords,

restrict 127.0.0.1

Another thing we should make sure, that if the NTP server is disconnected from the internet, the server provides time from its local system clock,

server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

server says that the local system clock is the timeserver, fudge is the keyword to fake the local server with a high stratum. So, when the server is connected with the internet it will still use the l33t timeserver who has the lowest stratum.

After configuration we must [re]start the NTP server,

$ service ntpd start

check if NTP server is configured properly, it’ll show all the time servers,

$ ntpq -p

To configure Linux Client just add the NTP servers IP with following,

$ ntpdate -u xxx.xxx.xxx.xxx

To check which stratum is running on NTP server,

$ ntptrace

And that’s all. If I miss something important please feel free to add in the comment.

Static IP Address Fedora 16

This is another quick post on Fedora 16. Setting up static IP address is little different from Debian based distros and also there is a slight change from the previous Fedora distros.

first check the network devices installed in the machine

$ ifconfig

for me, there is a device installed p1p1 (except lo). Now, create a network script file.

$ vi /etc/sysconfig/network-scripts/ifcfg-p1p1

and fill it up with your hardware and connection detail.

DEVICE=p1p1
BOOTPROTO=static
IPADDR=10.10.10.100
NETMASK=255.255.255.0
GATEWAY=10.10.10.2
HWADDR=94:0C:6D:86:95:F7
DNS1=8.8.4.4
DNS2=203.112.72.5
ONBOOT=yes
NM_CONTROLLED=no

I think most of the attributes are understandable. however, NM_CONTROLLED means Network Manager Controlled.

now, restart the network

$ sudo /etc/init.d/network restart

$ ping yahoo.com

….tada!!!

Enable SSH on Fedora 16

A very quick post, I needed after a fresh Fedora 16 install.

Enable sshd service.

$ systemctl enable sshd.service

start sshd service

$ systemctl start sshd.service

check sshd status if needed.

$ systemctl status sshd.service

restart sshd service, when needed.

$ systemctl restart sshd.service

stop sshd service and duck down 😛

$ systemctl stop sshd.service

well, make sure you have port 22 open.

$ system-config-firewall

….and that’s all for this quick note.

GNU screen – blessing for sys.admins

GNU screen is probably one of the most useful stuffs that makes me tension free and let my laptop to enjoy some hibernation time 🙂

GNU screen is terminal multiplexer, it lets admins to have multiple shell screens simultaneously. The most blessing part of it is, admins can leave anytime with any process running. It will keep doing its given task and then again can be used later from where admin left it. Such a wonderful app, eh?

installation process is just typical.

So like the famous GNU text editor emacs, it uses ctrl+a.

to detach from current window ctrl+a d

to kill current window ctrl+a k

switching between windows ctrl+a n & ctrl+a p

Every screen has an ID.

to get the screen back to work screen -r <screen id>

connect: Network is unreachable

So suddenly I was getting this weird error. Don’t know the exact reason. But some forum says it may happens because of having multiple NICs.

Anyway, then I checked $ sudo netstat -nr

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0

then run the following command

$ /sbin/route add -net 0.0.0.0 gw 10.10.10.1 eth0

and this time the result was different

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          0 eth0
10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0

….tada!!! It started working!