вторник, 22 декабря 2015 г.

Bye-bye, sysidtool, hello sysding

Often you want to have some simple tool to configure basic system settings after installation, such as ip settings, time zone settings, locales and so on. More important, installer also sometimes needs similar utility, which would run on first boot and initialize basic system parameters. For example, OmniOS runs /.initialboot script on the first boot. Solaris historically had sysidtool service, which read /etc/sysidcfg file and used it to configure zones or base system. Sysidtool also had ncurses-based interface to perform basic zone configuration.
The disadvantage of sysidtool is that it is a closed source tool, and you cannot fix it if you want it to do a bit more. So, we switched to sysding in OpenIndiana Hipster.
Sysding was originally written by Olaf Bohlen (Agnar at #oi-dev) to configure multiple illumos/Solaris zones. It doesn't have interactive interface, but has a set of utility functions to write configuration scripts. File /etc/sysding.conf is a simple ksh script, sourced by /lib/svc/method/sysding on . /lib/svc/method/sysding predefines some useful functions which can be necessary for initial configuration. Sample configuration file can look like
setup_timezone Europe/Moscow
setup_locale en_US.UTF-8
setup_user_password root '$5$+Fu+utqXFqU=$RD2LbFipqwKc2srNFYnVkda9U6K2pmMajvuR3iyHzR'
setup_interface PRIMARY v4 192.168.1.4/24
setup_route default 192.168.1.1
setup_ns_dns "stud.lan" "stud.lan notebook.lan" "8.8.8.8"

Using it, sysding will set timezone, locale, root password, network settings on first boot and reboot zone (or NGZ), because /etc/default/init was changed. It also cares about setting root password to 'NP' at first boot in zone if it's empty and you haven't specified one. Without this you wouldn't be able to "zlogin" to the zone. It can do a bit more. If you are interested, look at /lib/svc/method/sysding . If you want to have some customizations for your environment, create pull requests against https://github.com/OpenIndiana/sysding/, but don't forget two things: test your changes thoroughly and keep in mind that sysding was created to be a simple configuration tool.

среда, 7 октября 2015 г.

Userland incorporation in OpenIndiana Hipster and what does it mean for developer

Last Sunday we've published userland incorporation to /hipster-2015 repository. This was a feature long asked for by several users. It is generated by Jenkins on the build host and forces all packages generated by oi-userland build (excluding illumos-gate-provided packages and kvm) to be the latest. As we publish this incorporation on each oi-userland rebuild, you can force your system to go to specific point in the future. For example:

$ pkg list -avf pkg://openindiana.org/consolidation/userland/userland-incorporation
FMRI                                                                         IFO
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151006T203007Z ---
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151005T203207Z ---
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151004T163056Z i--
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151004T150924Z ---
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151004T132353Z ---
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151004T122350Z ---
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151004T095518Z ---
....

We can see that this system has incorporation with  20151004T163056Z  timestamp installed and two more recent versions are available. So I can do something like:
$ sudo pkg update -v \
pkg://openindiana.org/consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0:20151005T203207Z

to move to the 5th October  package versions.

Why developers doesn't like incorporations (and IPS generally)? Because it doesn't allow you to do what you want with your system. For example, you can't install another package version.
In case of userland incorporation, you have some freedom. You can just uninstall it (and entire, as entire depends on userland-incorporation). But you can get issues during new zone setup if your NGZ misses entire. So, you have three options:
  • stay with old entire, which doesn't depend on userland-incorporation (pkg freeze it);
  • uninstall entire and userland-incorporation;
  • use facets to relax incorporate dependencies.

Let's look on the last option more attentively. For example, I'm going to experiment with new mesa. But userland-incorporation has the following dependency:
$ pkg contents -m userland-incorporation |grep mesa
depend facet.version-lock.x11/library/mesa=true fmri=x11/library/mesa@10.5.9,5.11-2015.0.1.0:20150927T212600Z type=incorporate


As you see, it is marked by facet. So you can do
$ sudo pkg facet facet.version-lock.x11/library/mesa
FACET                                                            VALUE SRC
version-lock.x11/library/mesa                                    True  system
$ sudo pkg change-facet facet.version-lock.x11/library/mesa=false
$ sudo pkg install pkg://userland/x11/library/mesa
$ sudo pkg info mesa
             Name: x11/library/mesa
          Summary: The Mesa 3-D Graphics Library
         Category: System/X11
            State: Installed
        Publisher: userland
          Version: 11.0.2
           Branch: 2015.0.1.0
   Packaging Date: October  7, 2015 03:00:58 PM
Last Install Time: October  7, 2015 06:49:24 PM
             Size: 34.82 MB
             FMRI: pkg://userland/x11/library/mesa@11.0.2-2015.0.1.0:20151007T150058Z
      Project URL: http://www.mesa3d.org/
       Source URL: ftp://ftp.freedesktop.org/pub/mesa/11.0.2/mesa-11.0.2.tar.xz
But beware, if you would like to change facet back, you can't do it:
$ sudo pkg change-facet version-lock.x11/library/mesa=true
Creating Plan (Solver setup): /
pkg change-facet: Package entire must be uninstalled before the requested operation can be performed.
  Reject:  pkg://openindiana.org/entire@0.5.11-2015.0.2.1:20151003T221212Z
  Reason:  No version matching 'require' dependency consolidation/userland/userland-incorporation can be installed
Package x11/server/xorg/driver/xorg-video-ati must be uninstalled before the requested operation can be performed.
  Reject:  pkg://openindiana.org/x11/server/xorg/driver/xorg-video-ati@6.14.6-2015.0.1.0:20150927T212825Z
  Reason:  No version matching 'require' dependency x11/server/xorg@1.14.7-2015.0.1.0 can be installed
    ----------------------------------------
    Reject:  pkg://openindiana.org/x11/server/xorg@1.14.7-2015.0.1.0:20150927T184317Z
    Reason:  No version matching 'optional' dependency x11/library/mesa@7.4.4-2014.1.3.0 can be installed
      ----------------------------------------
      Reject:  pkg://userland/x11/library/mesa@11.0.2-2015.0.1.0:20151007T150058Z
      Reason:  This version is excluded by installed incorporation consolidation/userland/userland-incorporation@0.5.11-2015.0.2.0
      ----------------------------------------
    ----------------------------------------

.....


Firstly, you have to install mesa version offered by openindiana.org publisher.
Update (2016-01-26). If you use your host as test station, it's easier just to uninstall userland-incorporation. Now you can do this without touching entire. Just do
$ sudo pkg change-facet facet.require.consolidation/userland/userland-incorporation=false
$ sudo pkg uninstall userland-incorporation

пятница, 2 октября 2015 г.

Don't think for me...

I really like OpenVZ and Proxmox. But what I hate is programs which try to think for me. For example, we have /var/lib/vz and subdirectories on ZFS. If by chance it was not mounted on system startup, OpenVZ creates subdirectories in /var/lib/vz/template. Next time ZFS filesystem containers/vz/template just will not be mounted on non-empty /var/lib/vz/template. And you discover it only during runtime. If OpenVZ just threw error on startup, it would be better. If ZFS just silently mounted filesystems over non-empty directories (as usual mount does), it would be better. But two subsystems try to think for me and make my life better. I hate programs being so smart...

четверг, 23 июля 2015 г.

On attaching zones and linked images

Recently updated IPS was added to OI and it caused some inconvenience to developers. The most annoying "feature" is caused by publisher check: NGZ first several publishers should be the same as GZ's publishers and their stickiness should match.
OK, I've set the publishers according to this rule, but today was surprised by fact that I can't longer set openindiana.org publisher to be non-sticky in NGZ. I've set openindiana.org publisher to non-sticky in GZ, NGZ, but still IPS complains on "pkg install":

pkg install: Invalid child image publisher configuration.  Child image publisher
configuration must be a superset of the parent image publisher configuration.
Please update the child publisher configuration to match the parent.  If the
child image is a zone this can be done automatically by detaching and
attaching the zone.

The parent image has the following enabled publishers:
    PUBLISHER 0: openindiana.org
    PUBLISHER 1: userland (non-sticky)

The child image has the following enabled publishers:
    PUBLISHER 0: openindiana.org (non-sticky)
    PUBLISHER 1: userland (non-sticky)
    PUBLISHER 2: hipster-encumbered

I've rechecked. Both GZ and NGZ had  openindiana.org publisher set to non-sticky.  I've followed IPS advice - detached and attached zone. And zone attach -u failed with the same error.

OK. Time for black magic. I've unset userland publisher. Set GZ and NGZ publisher list only  to openindiana.org (non-sticky). The same issue - IPS doesn't see that GZ publisher is non-sticky now. So, I concluded that  information about parent image is recorded or cached in some local zone configuration file. Looked at zone's /var/pkg and found /var/pkg/linked/linked_ppubs, which listed [["openindiana.org", true], [userland, false]]. I changed this file to [["openindiana.org", false]] and after that could attach zone.

I think I still doesn't understand as linked images work (or should work), but they are starting annoying me...

вторник, 31 марта 2015 г.

Hipster 2015.03 is here


We released our last snapshot ISO almost half a year ago. I believe, you want something new. You'll get it. New ISOs were just uploaded to dlc server. Let's see what has changed.

First of all, most evident changes were made in desktop area. We've updated Xorg server and libraries, which allowed us to incorporate some important security fixes from Oracle x-s12-clone and Debian Xorg. Also we've moved much more closely to Gnome 2.32. Most packages were updated to this level, excluding packages which either have a lot of specific patches (like gdm) or just dropped some significant functionality (like cheese, which dropped HAL support in version 2.32). Not everything has gone smoothly. We had to drop trusted desktop support during update. I believe nobody seriously used it under OI. The most annoying thing is that updated Xorg and Intel driver require some DRM updates, which are still not ready. So, if you have Intel video card, either pkg freeze X-incorporation and xorg, or use vesa driver.

General system changes

All Sun Studio-compiled C++ libraries were removed from the system. The libraries were published in their current form to http://dlc.openindiana.org/c++-libs/, so you can grub necessary libraries and LD_PRELOAD them or use in alternative path if necessary. All X/g++/Y packages are renamed to X/Y and moved from /usr/g++ to /usr. We continue delivering system/library/c++/sunpro for the foreseeable future.
Text installer was changed to install OI on EFI-labeled disk by default. Note, in this case the entire disk is erased. If you want to install OI on MBR-labeled disk, choose partitioned install.

Desktop software and libraries

  • A lot of desktop libraries were updated
    • GTK2 is updated to 2.24.27
    • libdrm is updated to 2.4.58
    • libX11 is updated to 1.6.2, xcb support is enabled in libX11
    • xf86-video-ati driver updated to 6.4.16
    • nvidia proprietary driver was updated to 340.76
    • Mesa is updated to 10.5.1
    • Xorg is updated to 1.12.4. This requires updating xorg drivers and modules. OI-shipped modules will be updated automatically, but if you use VirtualBox, you'll have to update your guest additions to at least 4.3.22 version.
    • Glib2 is updated to 2.43.4
  • Enlightenment 0.19.3 is added as alternative desktop environment
  • fontconfig was updated to 2.11.1
  • libid3tag and libmtp were imported from SFE, gmtp is added
  • rdesktop is updated to 1.8.3
  • transmission is updated to 2.52
  • XScreensaver is updated to 5.32
  • gnome-commander is updated to 1.4.5
  • QT 4.8.6 is added
  • emacs is updated to 24.3
  • Input Method Selector was added from upstream input-method gate. Bug in svc:/application/desktop-cache/input-method-cache:default service preventing correct input methods functioning in recent OI was fixed. In fact, gtk input modules cache was moved from /etc/(amd64/)gtk-2.0/gtk.immodules to /usr/lib/(amd64/)gtk-2.0/2.10.0/immodules.cache and service has to regenerate these cache files in new locations . So, after update you can safely remove /etc/(amd64/)gtk-2.0/gtk.immodules.

Development tools

  • Subversion is updated to 1.7.19
  • SQLite is updated to 3.8.8.3
  • Python 3.4 is updated to 3.4.3
  • Binutils are updated to 2.25
  • OpenBLAS 0.2.13 is added
  • Mercurial is updated to 3.3
  • Ruby 1.9.3 is added
  • Ruby 1.8 is marked obsolete, all OI software is switched to Ruby 1.9.3
  • Ruby 2.2.1 is added
  • Curl is updated to 7.39
  • libncurses.so links are moved to /usr/lib(/amd64)
  • gawk is updated to 4.0.2, this fixes issues with pkgsrc bootstrap
  • MPICH is updated to 3.1.3
  • Sun Studio indent in /usr/bin was replaced by GNU indent. Old one is preserved in /opt/sunstudio12.1/prod/bin/indent

Server software

  • A lot of packages were updated, including apache 2.4, php 5.4, php 5.5, postgresql 9.3, samba 3.6, mariadb 5.5
  • PostgreSQL 9.4 is added
  • PostgreSQL 8.4 is marked obsolete
  • ISC dhcp server is updated to 4.2.7
  • BIND DNS server is updated to 9.9.6-P2
  • rsyslog is updated to 7.4.10
  • NTP is updated to 4.2.8p1

As always, we are proud to deliver to you latest illumos-gate bits.

There's also a lot of security fixes and small bug fixes.

Of course, I had more ideas than spare time, so some of them were not implemented. We still don't have PHP 5.6 and our OpenOffice package still doesn't work with XML-based document formats. I've looked at replacing cpp with one based on Schilix version, but unfortunately I found it to be not always compatible with Sun cpp. So, I've chosen preserve the status quo and we still deliver Sun cpp. I still would like to see postfix as first class MTA in OI.

I'd like to share some more ideas, which attract me now. First of all, we consider further updating of Xorg and other former xnv components (libXfont, freetype). GCC update is also on the roadmap. Our old samba and cups versions, dependency on python 2.6 and ageing Perl 5.16 make me sad.  Of course, I'd like to see PHP 5.6 in oi-userland and perhaps even look at hhvm.