Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Saturday, July 22, 2023

Linux kernel drivers, initrd vs initramfs

The bootloader will load the kernel. This could be GRUB or BCD for windows

 Initrd - meant initial ram disk. Initramfs - initial ram file system.
It has most drivers built as modules, so you dont waste all the ram building drivers in the kernel.
It is big 50MB and contains all drivers. 

initrd - is compressed image. Zstandard compressed data/ ASCII cpio archive
initramfs - uncompressed for older pcs.

It is called initrd in many places although it is a initramfs, particularly in boot loaders, as for them it is just a BLOB. The difference is made by the OS when it boots.

dmesg

systemd-analyze 
systemd-analyze plot > plot.svg

/var/log/rc.log --- /etc/rc.conf - log=yes

Startup finished in 2.840s (kernel) + 11.378s (userspace) = 14.218s graphical.target reached after 11.364s in userspace

Userspace will be since systemd (init) starts to load services. 
sudo systemctl disable/enable/start/stop NAME_OF_SERVICE

journalctl --disk-usage
journalctl --vacuum-size=50M
journalctl --verify   (verify integrity)


/etc/systemd/journald.conf
SystemMaxUse=50M

Drivers

To use hw (video card, usb device etc) we need 1 or 2 parts depending on the hardware.
1. a device driver that provices software APIs to the kernel to interact with the hardawre and is os kernel specifici - windows/linux, or even different kernel versions, when some major APIs change.

Drivers can be built into the kernel or dynamic loaded. Most are dynamic for the flexibility.
Then there are in tree and out of tree drivers.
If the driver is open source it gets included into the kernel tree and gets re-compiled with every kernel version. If its not open source and its proprietary code, then you have to compile it with every new kernel you install.

Drivers depend on kernel APIs - so if the driver is added to the kernel-tree, when a developer changes the APIs they also update the drivers.
If some older drivers depen on interfaces or APIs that get dropped out the kernel, then they will stop working, and you need to buy new hardware.
Kernel developers work for free and the hardware manufacturers only support old hardware for a few years until they release new hw. None of them will be willing to update the software of old hw.

To some extent this also happens with Windows, as some older hw might only have drivers up to a specific windows kernel versions, lets say only up to windows xp.
But if the driver is proprietary and out the tree, then it will stop working with new kernels - unless someone updates the code, which rerelly happens - so you need to buy new hardware if you want to use the new kernels.

DKMS - Dynamic Kernel Module Support - need to run as root, will automate the rebuild process of external driver modules with every kernel update. 
You need to install it and configure it. --- sudo install dkms

Kernel driver is the one that is bound to the hw.
Kernel modules are different drivers that can be used, if different functionality is required. NVIDIA might provide multiple drivers for different hw accelerated stuff, directx etc.

Modules = driver /usr/lib/modules/6.1.0-15-amd64/

lsmod
modprobe
insmod
rmmod
sudo modinfo  --- module params ex usbcore --- old scheme, new scheme - read first 8bytes or 64bytes to know the speed

Proprietary out-of the tree drivers - Video drivers: 
xserver-xorg-video-intel 
xf86-video-intel 
xf86-video-amdgpu 
nvidia nvidia-utils

sudo apt install xserver-xorg-video-intell

This package provides the driver for the Intel i8xx and i9xx family of chipsets, including i810, i815, i830, i845, i855, i865, i915, i945 and i965 series chips.


2. a firmware - is a piece of code uploaded to the hw chip
In windows the driver includes both of them.
In linux it depends on the open-source license. So in most cases there is a open-source driver included in the kernel -tree and then you might need to get a firmware from the manufacturer.
Most of the time there is a package that it includes it, if not in the main dristo then in a separate repository like nonfree 

In dmesg you can see a driver request a firmware.
Some hw some the firmware written into the chip's memory, and you can make a firmware upgrade, or some hardware use dynamic firmware that is loaded at boot time into the hardware, for ex wifi cards from intel load the firmware dynamic. to install it you use the package iwlwifi - intel wifi lan driver. Even though its called driver it actually contains a firmware
/usr/lib/firmware
The firmware is loaded by udev and sent to the driver - that is a kernel module.
Most of the time kernel modules are drivers. But they can provide other functionality as well, usb stack, network stacks etc.

udev - device manager
DBUS - inter process comunication protocal

***** udevadm monitor
udev rules (1) /lib/udev  --- (2) /etc/udev  --- /usr/lib/firmware

For ttyUSB0 --- sudo usermod -a -G dialout $USER

Edit apt sources and add contrib non-free  
apt search firmware

some example of packages with firmware:
linux-firmware
firmware-linux-free
firmware-linux-nonfree
linux-firmware-nonfree
firmware-iwlwifi

Add driver at debian install time:
Search here:
https://www.debian.org/distrib/packages#search_contents
Put contents in /dev/sdc1/drivers and mount to /lib/firmware
/lib/firmware/iwlwifi-3945-2.ucode

mkdir /lib/firmware
mount /dev/sdc1/drivers /lib/firmware


https://wiki.ubuntu.com/Kernel/Firmware


How to recompile the kernel to stop using an initrd - for much faster boot

sudo apt install dpkg-dev
echo "deb-src https://pkgmaster.devuan.org/merged stable main non-free-firmware" | sudo tee -a /etc/apt/sources.list
*copy first line and add in front deb-src

sudo apt update
apt source linux
cd linux-6.12.73   --- whatever version it will be current when you do it
cp /boot/config-$(uname -r) .config    --- we copy current config

OPTIONAL

 If you have a slow pc but have access to a faster pc for compile 
tar -cf custom_kernel.tar linux-6.12.73/
Boot an arch linux live image on a fast pc.
pacman -Sy base-devel bc kmod cpio libelf pahole openssl
tar -xf custom_kernel
.tar -C /root

mount -o remount,size=12G /run/archiso/cowspace
If you get errors you might need
make clean --- deletes all compiled files, you start over from 0
make oldconfig ---- addes changes to .config


make menuconfig
make -j$(nrpoc) or make -j16


Build kernel
============
sudo apt install gcc flex bison libncurses-dev libssl-dev
make xconfig
make menuconfig
/usr/src/linux/.config


make localyesconfig --- local loaded modules --- built-in kernel
make localmodconfig --- local loaded modules --- built as modules
make allnoconfig --- minimum kernel :)
+ [sata + ata + scsi] (ahci - lspci -k) si fs(ext4)

time make -j4 // j3 =j4
make modules
sudo make modules_install --- /usr/lib/modules/xx.xx.xx/
sudo make install --- /boot/

https://www.kernel.org/  download kernel
You can make it in ramdisk - and copy final back to hdd for further changes.

make mrproper- The make mrproper command cleans up any leftover files from previous kernel builds in your source directory. It also wipes out any . config files in your source directory. If you want to do a new clean build.
Boot a Linux MX image to detect all drivers.
ADD all drivers loaded either built into kernel or as modules.

lsmod > /tmp/mylsmod
 make LSMOD=/tmp/mylsmod localmodconfig
Press / and search for ex USB_SERIAL_CONSOLE  for usb to serial driver - and it shows the path where to find it.

TO disable initrd, disable:
General Setup -> at the bottom - initrd imitramfs support
Need to have added as drivers in kernel ext4 and sata/ahci or nvme drivers

For iwlwifi driver to work when built as driver into the kernel you need  to set:
CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
Linux Kernel Configuration
└─> Device Drivers
└─> Generic Driver Options
└─> Firmware loader
└─> Force the firmware sysfs fallback mechanism when possible

No rule to make target 'debian/certs/debian-uefi-certs.pem'
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS



Hardware info

lspci -k
sudo dmesg | grep driver --- to view version
journalctl

sudo dmidecode --mb info
sudo dmidecode -t 16 (max) / 17 (modules) - ram
lm-sensors --- temps/fan/voltages --- sensors
lsusb
lsblk
lshw
uname

lscpu    --- cat /proc/cpuinfo
uname -a --- cat /proc/version --- linux distro version
cat /etc/os-release       --- distro - debian 11 bullseye

Check OpenCL drivers

You can run clinfo (sudo apt install clinfo ) and if it says 0 devices

sudo apt install intel-opencl-icd 

Thursday, June 1, 2023

Manual install devuan or from scratch

Boot  Arch linux image from USB. Use rufus to burn it windows,  or dd in linux.
dd if=~/Download/arch.iso of=/dev/sdb bs=4M status=progress oflag=direct

* Install base packages distro

Wifi - internet
iwctl station wlan0 connect SSDID_name

Partition
lsblk
cfdsik -> sda1 100-200MB efi , sda2 20GB
mkfs.ext4 /dev/sda2
mount /dev/sda2 /mnt/linux

Install:
- from arch: pacman -Sy debootstrap
- from debian: apt install debootstrap

Install debian: debootstrap --arch amd64 stable /mnt/linux
Install devuan: debootstrap --arch amd64 stable /mnt https://pkgmaster.devuan.org/merged

mount --bind /dev /mnt/dev

mount --bind /dev/pts /mnt/dev/pts

mount --bind /proc /mnt/proc

mount --bind /sys /mnt/sys

mount --bind /run /mnt/run

chroot /mnt /bin/bash --login  (for proper $PATH)

arch-chroot --- does auto mount/bind those but doesn provide the login paramer and $PATH will be wrong so dont use it.

Apt remove cron-daemon-common

Apt install systemd-standalone-sysusers devuan-keyring ca-certificates apt install cron-daemon-common

Dpkg --configure -a Apt install openrc


apt install nano apt-file

/etc/fstab
/dev/sda2 / ext4 rw,relatime 0 1


Set Time Zone:

ln -sf /usr/share/zoneinfo/Europe/Bucharest /etc/localtime
date

apt install debconf
dpkg-reconfigure tzdata
dpkg-reconfigure locales

apt install sudo
passwd   --- create password for root or you cant login after reboot

adduser username
usermod -aG sudo user

Kernel
apt install linux-image-amd64  (it install the latest kernel its meta pkg)
apt install grub-efi-amd64

or use your kernel if you have one
copy /boot/vmlinuz 
copy /usr/lib/modules/uname -r/ /mnt/linux/.... same path

Install GRUB -- I sugest to install it on the uefi partition so systems updates dont mess it
mount /dev/sda1 /boot/efi
grub-install --boot-directory=/boot/efi --efi-directory=/boot/efi --target=x86_64-efi
ln -s /boot/vmlinuz-xxxx vmlinuz
ln -s /boot/initrd-xxxx initrd
nano /boot/efi/grub/grub.cfg
menuentry 'Linux' {
    set root ='hd0,gpt2'
    linux    /boot/vmlinuz root=/dev/sda2 ro
    initrd    /boot/initrd
}

WIFI

nano /etc/apt/sources.list - add non-free-firmware
apt update
apt install iproute2 net-tools rfkill iwd ifupdown
#apt install firmware-misc-nonfree    --- othter wifi boards - realtek asus
apt install firmware-iwlwifi    --- mostly intel wifi boards
apt install pciutils lsutils

apt install iwd (alternative is wpa_supplicant) 
Enable dhcp client + dns
/etc/iwd/main.conf
[General]
EnableNetworkConfiguration=true
/etc/resolv.conf
nameserver 1.1.1.1

nano /etc/hostname
nano /etc/hosts    - 127.0.0.1 hostname

/etc/network/interfaces
auto lo
iface lo inet loopback


Reboot

* Video system
You can install here a desktop/login manager - gdm3, lightdm etc OR just use xinit without an GUI interface
apt install xfce4   --- it will install xorg as a requirement

Note on devual bare xinit doesnt work, it requires to many manual changes just install lightdm.

Change console FONT size

Go to directory /usr/share/kbd/consolefonts - and ls to see all available fonts

then setfont iso01-12x22.psfu.gz 
or any other fonts you like from there

dpkg-reconfigure console-setup

Change active Display Manager:
sudo dpkg-reconfigure gdm3
sudo dpkg-reconfigure lightdm


https://www.linuxfromscratch.org/

ArchLinux - https://wiki.archlinux.org/title/Installation_guide

https://wiki.debian.org/Debootstrap

Saturday, April 1, 2023

Storage partitions under linux - swap - ramdisk

Partitions

gparted, cfdisk, lsblk, df -h

mkfs.ext4 /dev/sda2 , mkfs.ntfs etc --- format the partition  

sudo fdisk -l     --- view disks
lsblk -f     --- view disks and filesystems
sudo blkid - partition UUID
df -h
df -ih --- innodes

sudo tune2fs -l /dev/sda8
sudo tune2fs -m 1 /dev/sda8
Reserved blocks for root - so when you run out of space you can still login. Set default to 5% of partition size..
*** merge online cu partitia mounted = ok
After you rezise partition size with cfdisk you need to also resize the filesystem:
sudo resize2fs /dev/sdb

Mounting

/mnt : Mount point for a temporarily mounted filesystem - should not have subfolders
/media : Mount point for removeable media

These 3 are required for chroot, 

are interfaces to kernel data structures and not real files on hdd - they stay in ram.

Can check with the command findmnt /sys
/dev - file devices
/proc - running process table + some system info same as sys left for backward compatibility 
/sys - hardware information

/run - virtual ram runtime for udev - needed before /var/run is mounted

/etc - system config files - before changes would be done in source code and recompile
means etcetera - as in the rest of the files of a system

/var - variable stuff like logs
/opt - optional software not installed with system package manager
/usr - libraries, binaries, manuals - all the software

/etc/fstab --- automount at start-up - should at least have /

mount -a --- mount all from fstab

UUID=14D82C19D82BF81E /mnt/d auto nosuid,nodev,nofail,x-gvfs-show 0 0

1. partition id
2. mount point
3. partition type 
4. options

nosuid - specifies that the filesystem cannot contain set userid files. This prevents root escalation and other security issues.
nodev - specifies that the filesystem cannot contain special devices (to prevent access to random device hardware).
nofail - removes the errorcheck.
x-gvfs-show - show the mount option in the file manager. If this is on a GUI-less server, this option won't be necessary.
0 - determines which filesystems need to be dumped (0 is the default).
0 - determine the order in which filesystem checks are done at boot time (0 is the default).

mount /dev/sda1 /mnt

NTFS partition from windows closed without proper shutdown ---> ntfsfix /dev/sda1
If that doesn't work you need to boot a windows os and it will fix it. ntfs is not opensource.

See mounted filesystems and space usage : 
du -sh -- file/folder size

INODES on linux

The file system has an inode table - every file and directory on a disk requires one inode entry.

To have more inode you need to re-format the filesystem and reserve more space for the inode table.

Other filesystems dont allocate a fixed inode table like XFS, JFS, BtrFS or ZFS.

On ext3 and ext4 - the most common used filesystems on linux you get by default 1 inode for every 16kb of space.
using the -i flag to mkfs.ext4 to specify the bytes:inode ratio. 

This problem is most common on servers where you can get a lot of small files.
Apache’s disk cache system (mod_cache_disk) - creates a lot of small files - htcacheclean should clean it.

Swap

Instead of storage swap partition you can have a swap partition in ram.
Of course this will not work for hibernation.

Swap file

dd if=/dev/zero of=/swapfile bs=1M count=3064 status=progress
chmod 600 /swapfile
mkswap -U clear /swapfile
swapon /swapfile
Add it to fstab
/swapfile none swap defaults 0 0 >> /etc/fstab

Ram swap

When ram gets full, data is moved and compressed into this z swap partition.

To enable it, open a cmd and create these files :

sudo su
echo zram > /etc/modules-load.d/zram.conf
echo 'options zram num_devices=1' > /etc/modprobe.d/zram.conf
echo 'KERNEL=="zram0", ATTR{disksize}="512M",TAG+="systemd"' > /etc/udev/rules.d/99-zram.rules

touch /etc/systemd/system/zram.service

[Unit]
Description=Swap with zram
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStartPre=/sbin/mkswap /dev/zram0
ExecStart=/sbin/swapon /dev/zram0
ExecStop=/sbin/swapoff /dev/zram0

[Install]
WantedBy=multi-user.target

sudo systemctl enable zram
reboot 

swapiness - sudo nano /etc/sysctl.conf

The default value is 60."
0 - does not disable swap

To use a swap partition instead

# mkswap /dev/sda5
# sync
# swapon /dev/sda5

Ram-disk

For windows :
The only free software that allows for more then 4gb is ImDisk Toolkit

For linux  :
First check how much ram you got : free -h
Make a new directory and mount the ramdisk :

sudo mkdir /mnt/ramdisk
sudo mount -t tmpfs -o size=10g tmpfs /mnt/ramdisk
tmpfs - is mounted in ram
To make it available every time you boot edit /etc/fstab :

tmpfs       /mnt/ramdisk tmpfs   nodev,nosuid,noexec,nodiratime,size=1024M   0 0

Monday, December 12, 2022

Linux autologin + Sudo with yubikey

Authentification is done by PAM - Pluggable Authentication Modules
Passwords are stored in /etc/shadow

passwordless login with yubikey on linux

HMAC-SHA1 Challenge-Response is specific for Yubikey only.

yubiko-pam - is yubikey-specific and relay on it's specific features, 
pam-u2f - is generic and works with any keys supporting u2f and/or fido2.

auth required pam_u2f.so cue interactive

 Set authentification with yubikey off-line with Challenge-Response so you dont have to type a password just touch the keyyubikey

install yubico pam package
# install chalenge to slot 1
# generate challenge file for slot 1
ykpamcfg -1 -v
man ykpamcfg

/etc/pam.d/common-auth
#[success=1 new_authtok_reqd=ok ignore=ignore default=die]
auth required   pam_echo.so "Touch the key"
auth  sufficient      pam_yubico.so mode=challenge-response authfile=/home/user/.yubico/challenge-xxxx

Install xscreensaver if you want to lock the screen

Autologin with desktop/login manager

autologin with lightdm:
/etc/lightdm/lightdm.conf -- uncomment the lines
autologin-user = sorin
autologin-user-timeout = 0
*Note if you cannot login with the user but you can with root check if you ran out of space. (df -h)

Autologin with systemd directly

Agetty is called by /bin/INIT and this calls /bin/login 

/etc/systemd/system/getty.target.wants/getty@tty1.service
add -a my_username

ExecStart=-/sbin/agetty -a username - $TERM
remove the login options -o '-p -- \\u'

To start X11 with xfce
~/.bash_profile
if [ -z "${DISPLAY}" ] && [ $(tty) = /dev/tty1 ]; then
  startxfce4
fi

Tuesday, November 8, 2022

Mouse latency response time in linux

 lsusb

Find your mouse. 

Bus 001 Device 003: ID 1a7c:0191 Evoluent VerticalMouse 4

sudo usbhid-dump -s 1:3 -f -e stream

From the manual we get what the parameters mean:
man usbhid-dump
 -s =bus[:dev]
Replace -s 1:3 with the Bus Device for your device

Output is in the form
BUS:DEVICE:INTERFACE:ENTITY TIMESTAMP

.001:003:000:STREAM             1667902900.482858
 00 00 00 FC FF 00 00

.001:003:000:STREAM             1667902900.490787
 00 00 00 FE FF 00 00

.001:003:000:STREAM             1667902900.498863
 00 01 00 FC FF 00 00

Timestamp is in seconds before the point and subdivisions of seconds after the point.
First 3 digits milliseconds next 3 digits microseconds.
So the difference between two events is in my case is 8ms.
Note the speed dpi choise does not affect this. This is the maximum physical transmit speed.

usbhid-dump will disconect you mouse/device.
if you mess up just remove the receiver and plug-it back in or wait 60 seconds.

It is a WiFi Evoluent Vertical Mouse 4.

Configure Mouse Buttons X11
xinput
press a button and run in terminal to find the number associated with each button
xinput query-state id-nr
or
xev | grep button


xinput set-button-map "Kingsis Peripherals Evoluent VerticalMouse 4" 1 2 3 4 5 6 7 9 2 8

90-evoluent.conf /usr/share/X11/xorg.conf.d/conf
Section "InputClass" Identifier "Evoluent VerticalMouse 4" MatchUSBID "abcd:0123" # input id: 1 2 3 4 5 6 7 8 9 10 Option "ButtonMapping" "1 2 3 4 5 6 7 9 2 8" EndSection


Thursday, March 3, 2022

Linux Python

 To install a custom version of python, download and compile from here:

https://www.python.org/ftp/python/

MuEditor - requires python < 3.9 + install libsqlite3-dev before compiling 

python3.8 -m venv mu-editor              (create a virtual env in the mu-editor folder)
cd mu-edit/bin
source activate                                       (activates in console the virtual env)

pip install mu-editor


To install python apps you use pip

Pip aplications: https://pypi.org/project/mu-editor/

Make shorutcuts for python apps while in virtual env

pip install shortcut
shortcut mu-editor

Wednesday, January 12, 2022

Install scanner in linux - Brother DCP-7030

Scanner in LINUX - SANE - Scanner Access Now Easy


sudo sane-find-scanner  *** list connected scanners
lsubs *** view usb connected devices ( sudo apt install usbutils )

scanimage -L  *** check if sane detects your scanner
SANE_DEBUG_DLL=3 scanimage -L
load: dlopen() failed (libusb-0.1.so.4: cannot open shared object file: No such file or directory)

To fix it:
apt search --names-only libusb-0.1
rezult: libusb-0.1-4/stable 2:0.1.12-32 amd64 
sudo apt install libusb-0.1-4 

Open the file -> /etc/sane.d/dll.conf 
and add one line with 
brother3
!!! note when you run scanimage -L it will check every record, so just delete all records and leave your printer name

Create a new file with your printer name and add it.
brother.conf
firmware /usr/lib64/sane/libsane-brother3.so.1.0.7
usb 0x04f9 0x01ea

!!! you get vendor id from lsusb or sane-find-scanner

You might just use the install package to install the rules file automaticly instead.
/etc/udev/rules.d/60-myprinter.rules
# Brother scanner
ATTRS{idVendor}=="04f9", MODE="0666", GROUP="scanner", ENV{libsane_matched}="yes"
!!! change idVendor - with your printer vendor id

Brother DCP-7030
udev rules -> brother-udev-rule-type1-1.0.2-0.all.deb
driver -> brscan3-0.2.13-1.amd64.deb

reboot
Things I did that i don't think are necesary:
sudo usermod -a -G myuser scanner
Copy the driver files under /usr/lib64/ to /usr/lib/.

For printing
Install cups, and brlaser (driver)
then go to localhost:631 - adminstration - add printer - done

Friday, September 10, 2021

Software management on linux

Debian has the largest package library of software.
It uses a package format .deb  that are installed by the dpkg app. 
In order to manage dependencies and download automatically there is apt (Advanced Packaging Tool).

Synaptic is a graphical package management tool based on GTK+ for apt.
Apt commands in the console.

apt installapt-get installInstalls a package
apt removeapt-get removeRemoves a package binaries
apt purgeapt-get purgeRemoves package with configuration and user data
apt autoremoveapt-get autoremoveRemoves unused dependencies
apt upgradeapt-get upgradeUpgrades all upgradable packages
apt updateapt-get updateRefreshes repository index / package list
apt full-upgradeapt-get dist-upgradeUpgrades packages with auto-handling of dependencies
apt searchapt-cache searchSearches for the program
apt showapt-cache showShows package details
apt listLists packages with criteria (installed, upgradable etc)
apt edit-sourcesEdits sources list

You can use synaptic  as a GUI for apt.

apt autoremove packagename --purge  - remove binaries, config, user data and dependencies - complete uninstall

apt list --installed | grep packagename - check if you have some package installed

Note: - purge only removes system wide configuration from the /etc folder.
It does not remove user profiles that are stored in the home folder .config and .cache folders

To remove 32 bit architecture packages and remove source repositories

sudo apt-get purge `dpkg --get-selections | grep ":i386" | awk '{print $1}'`
dpkg --remove-architecture i386
dpkg --print-foreign-architectures

https://ninite.com/ - package repository for windows.