Nearly all of our products ship pre-programmed and ready to boot out of the box. However, very few applications would find the unmodified filesystem suitable for their deployment and would require changes to be made in the filesystem. In many cases, it is easy enough to create a production process that can live-modify the filesystem; that is, a process to copy new or modified files to the filesystem where needed. However it can also be beneficial to completely remove what is on the disk and replace the whole file structure entirely. This could be due to the application needing a custom linux distribution or other operating system, or because the final disk layout needs a new partition scheme. Either way, it is important to have a production image that is clean from the start, one that does not have an excess of unneeded and/or temporary files. Additionally, it is highly desirable to have a process to recreate unique IDs on each platform; such as SSH keys, or unique machine identifiers.
When we prep an image for a release, we run through a number of steps. We clean temporary files that are auto-generated or that may have used during testing, and we also remove SSH keys and linux machine-id files. We then create md5sum file that can be used to verify the state of the filesystem during our production process. Finally, .dd and .tar files are created and compressed, and then are md5sum'ed themselves.
In order to accomplish this, we have created a simple script that will automatically run through various cleaning steps. Along the way it has shims in place to allow for additional scripts to be run. These shims allow for more product/application specific steps. An arbitrary set of prep scripts can be called before the image is finalized, as well as a second set of post scripts which are called before the final image is compressed and completed. For example, on i.MX28 based products, we use the prep scripts to ask what product the image is for, which will create or remove files on the filesystem as needed. Then we use the post scripts to generate a NAND, DoubleStore, or eMMC target output, which does its own compression on the output files as needed.
The script below can be used as-is, or modified to work in a specific application. Please note that this script is provided without any warranty. It is meant to be used with Debian based distributions on our products. There are a number of files that are deleted; please ensure that the script will not remove critical files, or make sure that any removed files (such as SSH keys) are intelligently re-created on first bootup of the newly imaged unit in your final application
The script can either be passed an existing .dd image file or a whole block device, as well as an optional name. If a block device, such as /dev/sdb, is passed, then the script will attempt to parse the partition table of the disk and create a .dd file that is from start of disk to the end of the final partition. All further operations then take place against the newly created .dd file, and the source disk is left alone. Note that if a .dd image file is passed to the script, rather than a block device, then that .dd WILL BE MODIFIED IN PLACE! If no name is provided, the current date is used.
A simple call to this script would look like:
'./prep_image /dev/sdb Company-prod-final-v1234'
This will create "Company-prod-final-v1234.dd" and all other created files will have that same prefix.
#!/bin/bash
#PREP_SCRIPTS=""
#POST_SCRIPTS=""
# This file is used to test and prepare and image for release. There are board
# specific tests that occur first, and then all of the generic test/prep
# occurs.
#
# 05/10/2017 - Remove /var/lib/dbus/machine-id file
#
# 02/22/2017 - Set /etc/passwd clearing to "x" to enable use of shadow
#
# 12/05/2016 - Added proper setup of /etc/hosts
#
# 11/21/2016 - Fixed comments to reflect what is done
#
# 08/23/2016 - Adjusted systemd detection to only look for networkd
#
# 12/14/2015 - Added detection of systemd, and adjusting /etc/resolv.conf
# Only attempt to adjust config options is /ts/config exists
#
# Run `prep_image <imagename>.dd[.bz2,.gz] | /dev/node`
# Current user must be root, since we are mounting
#
# Generic tests/preperation include:
# If $1 is dev node, create 64kbyte aligned image that is at least
# one sector longer than the last sector listed in fdisk
# Verify .dd is 64kbyte aligned
# fsck last partition
# Mount last partition
# Remove /etc/ssh/*key*
# Remove /etc/machine-id
# Remove /var/lib/dbus/machine-id (NOTE! Most Debian distros will NOT auto
# re-create this file! Some of our products will detect if it is not present
# and then create it on first bootup.)
# Comment "#" /etc/udev/rules.d/75-persistent-net-generator.rules
# Remove /etc/udev/rules.d/70-persistent-net.rules
# Remove /ts/*.vme*
# Remove /ts/init
# Remove /boot/*.sb
# Remove /boot/*.imx
# Remove /boot/*.vme*
# Clean up /root
# Only keep
# .bashrc
# .gtkrc-2.0
# .profile
# Remove /var/log/*, only files, leave folder tree intact
# Remove any packages in /var/cache/apt/archives (equiv. to apt-get clean)
# Set /root.version to current date
# Verify/set /etc/resolv.conf as a symlink to /dev/resolv.conf
# If using systemd, set up /etc/resolv.conf as symlink to systemd
# Remove root passwd if any, handles passwd and shadow
# If /ts/config exists, /ts/config lines are commented out, not CFG_XUARGS
# Set up /etc/hosts from /etc/hostname, use Debian defaults otherwise
# Run any board specific scripts in PREP_SCRIPTS
# Create /md5sums.txt of all of the md5sums of files in rootfs
# Create tarball of filesystem
# Zero FS for compression
# Unmount FS
# Post image production scripts in POST_SCRIPTS
# Compress image as bz2, and generate md5s for .dd and .dd.bz2
#
if [ $UID != 0 -o $# -eq 0 ]; then
echo "prep_image, a tool to prepare TS images for shipping. Runs various tests,"
echo "cleans up temporary files left in images, preps images, and outputs"
echo "compressed .bz2 and .md5 files"
echo "Note that running this on an image will result in image modification,"
echo "however running it on a /dev/node will first create an image, and"
echo "then modify it from there."
echo "This script assumes the last partition is the linux partition"
echo ""
echo "Usage: $0 </path/to/image/or/dev/node> [image_prefix]"
echo "image_prefix is used as the base name when pulling an image from dev node"
echo "If not provided, base name of +%d%b%y is used"
echo "Must be root when running this script, the use of \`mount\` is required"
exit 1
fi
if [ ! -e $1 ]; then
echo "Requested file does not exist!"
exit 1
fi
case $1 in
*".dd")
image_file=$1
;;
*".dd.bz2")
echo "bunzip'ing $1"
bunzip2 $1
if [ "$?" != "0" ]; then exit 1; fi
image_file=${1/.bz2/}
;;
*".dd.gz")
echo "gunzip'ing $1"
gunzip $1
if [ "$?" != "0" ]; then exit 1; fi
image_file=${1/.gz/}
;;
"/dev/"*)
EOD=`fdisk -l $1 | tail -n 2 | awk '$6 == 83 {print $3}'`
while : ; do
let EOD=$EOD+1
let EOD_byte=($EOD)*512
let x="$EOD_byte & 0xffff"
if [ "$x" -eq 0 ]; then break; fi
done
if [ $2 ]; then
image_file=${2}.dd
else
image_file="`date +%d%b%y`.dd"
fi
if [ -e $image_file ] ; then
echo "File already exists \"$image_file\" will not overwrite"
exit 1
fi
echo "Creating image file \"$image_file\" that is $EOD_byte bytes long from $1"
dd if=$1 of=$image_file bs=512 count=$EOD
if [ "$?" != 0 ]; then exit 1; fi
echo "Image file created from disk."
;;
*)
echo "File must be a .dd, .dd.bz2, .dd.gz, or /dev/*"
exit 1;;
esac
eval `stat -L -c "imgsize=%s" $image_file`
let x="imgsize & 0xffff"
if [ "$x" -ne 0 ]; then
echo "Error: Image is not aligned to 64kbyte"
exit 1
fi
linux_start=`fdisk -l $image_file | tail -n 2 | awk '$6 == 83 {print $2}'`
let linux_start=$linux_start*512
mkdir mount_point
echo "Performing fsck on last (linux) partition of disk image"
losetup /dev/loop0 -o$linux_start $image_file
if [ "$?" != "0" ]; then exit 1; fi
fsck /dev/loop0 -y
echo "Mounting last (linux) partition of disk image"
mount /dev/loop0 mount_point/
if [ "$?" != "0" ]; then exit 1; fi
echo "Removing temporary files, SSL keys, apt-get install files, etc."
rm -rfv mount_point/etc/ssh/*key*
rm -rfv mount_point/etc/machine-id
rm -rfv mount_point/var/lib/dbus/machine-id
rm mount_point/etc/udev/rules.d/70-persistent-net.rules
echo "#" > mount_point/etc/udev/rules.d/75-persistent-net-generator.rules
rm -rfv mount_point/ts/*.vme*
rm -rfv mount_point/ts/init
rm -rfv mount_point/boot/*.sb
rm -rfv mount_point/boot/*.imx
rm -rfv mount_point/boot/*.vme*
find mount_point/root/ | grep -v -e .bashrc -e .profile -e .gtkrc-2.0 -e "root/$" | xargs rm -rfv
echo "Removing log files (rm will error if no logs present)"
find mount_point/var/log/ -type f -print0 | xargs -0 rm -v
echo "Cleaning up apt packages and temp files (apt-get clean)"
rm -rfv mount_point/var/cache/apt/archives/* mount_point/var/cache/apt/pkgcache.bin mount_point/var/cache/apt/srcpkgcache.bin
vers=`date +%Y-%m-%d`
echo "Setting /root.version to $vers"
echo $vers > mount_point/root.version
if [ -e mount_point/etc/systemd/network/ ]; then
echo "Using systemd/networkd, setting /etc/resolv.conf as a symlink to /run/systemd/resolve/resolv.conf"
rm mount_point/etc/resolv.conf
ln -sf /run/systemd/resolve/resolv.conf mount_point/etc/resolv.conf
else
echo "Setting /etc/resolv.conf as a symlink to /dev/resolv.conf"
rm mount_point/etc/resolv.conf
ln -sf /dev/resolv.conf mount_point/etc/resolv.conf
fi
echo "Removing root passwd and enabling shadow"
awk -F':' '{if($1 == "root") {$2 = "x"; print $0} else {print}}' OFS=":" mount_point/etc/passwd > mount_point/etc/passwd.new
mv mount_point/etc/passwd.new mount_point/etc/passwd
awk -F':' '{if($1 == "root") {$2 = ""; print $0} else {print}}' OFS=":" mount_point/etc/shadow > mount_point/etc/shadow.new
mv mount_point/etc/shadow.new mount_point/etc/shadow
if [ -e mount_point/ts/config ]; then
echo "Commenting out all CFG_* opts in /ts/config except CFG_XUARGS"
sed -e '/CFG_XUARGS/!s/^CFG_/#CFG_/' mount_point/ts/config > fixed_config
mv fixed_config mount_point/ts/config
fi
echo "Setting up proper /etc/hosts file from hostname"
echo "127.0.0.1 localhost" > mount_point/etc/hosts
echo -n "127.0.0.1 " >> mount_point/etc/hosts
cat mount_point/etc/hostname >> mount_point/etc/hosts
echo "::1 localhost ip6-localhost ip6-loopback" >> mount_point/etc/hosts
echo "ff02::1 ip6-allnodes" >> mount_point/etc/hosts
echo "ff02::2 ip6-allrouters" >> mount_point/etc/hosts
echo "Calling Prep scripts"
for I in $PREP_SCRIPTS; do ./$I $image_file; done
echo "Creating md5sums.txt md5sums"
cd mount_point/
find . -type f \( ! -name md5sums.txt \) -exec md5sum "{}" + > md5sums.txt
cd ../
echo "Creating compressed tarball"
tar cf ${image_file%.dd}.tar -C mount_point/ .
bzip2 ${image_file%.dd}.tar
md5sum ${image_file%.dd}.tar.bz2 > ${image_file%.dd}.tar.bz2.md5
echo "Zeroing out free space in FS for better compression"
dd if=/dev/zero of=mount_point/zerofile
rm mount_point/zerofile
umount mount_point
if [ "$?" != "0" ]; then exit 1; fi
losetup -d /dev/loop0
if [ "$?" != "0" ]; then exit 1; fi
echo "Image file unmounted"
echo "Calling post scripts"
for I in $POST_SCRIPTS; do ./$I $image_file; done
echo "Compressing and generating md5s"
md5sum $image_file > $image_file.md5
bzip2 -9 $image_file
md5sum $image_file.bz2 > $image_file.bz2.md5